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Abstract 



The impact of the No Child Left Behind Legislation has left Career and Technical 
Education searching for methods and strategies to infuse academics into their curriculum 
to help students, through a rigorous and integrated educational experience, develop 
competencies required for higher education and ultimately, work. Unfortunately, there 
were pressures placed on states, districts, and schools to arrive at certain preset bench 
marks where ultimately, by 2014, all students would reach the 100 percent proficiency 
benchmark in state standardized achievement assessments. The NCLB along with IDEIA, 
partnered to make sure that all students with IEPs or 504 plans were held to the same 
standards regarding participation in these standardized assessments and also in the 
reporting process (Cornell University Law School website, n.d). 

This study looks at one part-time suburban Career and Technical Center and 
measures the impact of the software, Study Island, used to integrate academics into their 
Career and Technical Education programs. Specifically, 4Sight, PSSA, and NOCTI 
assessments were examined for effect and relationship with Study Island for students 
without IEPs and students with IEPs. Study Island is tutorial software that aligns with the 



state’s core curriculum and standards. 
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Chapter 1 
Introduction 

Historical Perspective 

From the Elementary and Secondary Education Act (ESEA) stemmed the No 
Child Left Behind (NCLB) Act of 2001 (No Child Left Behind Act of 2001 (Public law 
107-110), 2001). This created an era of accountability for all elementary and secondary 
schools in the United States. To keep pace with NCLB in Career and Technical 
Education (CTE) the Carl D. Perkins Act of 2006 initiated the CTE Improvement Act of 
2006 that mandated schools that received federal funding from Perkins to develop rigid 
academic and technical standards that improve skill sets of students who choose to attend 
Career and Technical Centers (CTCs) that will prepare them better to enter post- 
secondary institutions and the workforce (Carl D. Perkins Career and Technical 
Education Act of 2006 (Public Law 109-270) website, 2006). The Perkins Act ensures 
that CTCs will provide the necessary services that will enable all students to achieve at 
this level, to improve the tech prep initiatives, enable the state and localities to research 
best practices, provide professional development for teachers, administrators, and 
guidance personnel, to develop strong partnerships between postsecondary institutions 
and business and industry, and to develop a workforce that will keep the United States 
competitive in the global workforce. Under the authority of the NCLB Act, schools 
became accountable and are measured by achievement on standardized testing. There are 
annual publicized report cards and schools become at-risk of losing Title 1 and Perkins 
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funding if schools do not reach increasingly higher rates of proficiency on standardized 
state assessments (“Report card on the schools”, 2010). 

School implemented strategies. 

Although this study focuses on student learning, it is imperative to understand the 
legislation that drives school improvement which has led to strategies towards that end. 
As a result of pressures on CTCs to improve accountability, many strategies have been 
implemented to raise CTE students’ achievement on standardized assessments. 
Curriculum maps have been created that identify all of the standards and objectives and 
academic standards have been aligned with each objective as is appropriate. Initially the 
standards were broad, but academic anchors and eligible content sprouted from the 
standards to create specific and detailed examples of what may appear on the 
standardized assessment (SAS Standard Aligned Systems, n.d.). Professional 
development and in-services increased as a result of NCLB for CTC teachers to guide 
them in infusing academics into their CTE disciplines. Many numeracy and literacy 
strategies were integrated into theory lessons to ensure that students were receiving 
instruction in a variety of ways to account for the many learning styles of students. There 
became available grants for tutoring and as a result many schools were able to provide 
additional instruction after school hours. There were also grants for reading and math 
coaches to assist teachers in incorporating academics into their CTE curriculum. In the 
sending districts, schools were adding remediation courses for those at risk of not being 
proficient. 4Sight testing was incorporated in the sending districts and those districts that 
did not show adequate yearly progress (AYP) were mandated to adopt these tests and 
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give them four times a school year (Success for All Foundation, n.d.). CTCs were faced 
with the challenges of how to remediate without losing valuable technical theory and 
laboratory time. As a result, schools adopted strategies that were least invasive or that 
blended in with regular technical theory instruction. Many schools implemented 
Computer Aided Instruction (CAI) such as Carnegie, Plato, and Study Island. The school 
of this study implemented Study Island and the students get pulled from their technical 
discipline for 30 to 45 minutes per week. 

Not only do the students have to be proficient in academic skills as measured by 
the state academic assessment, but they also have to be proficient in their technical skills 
as measured by the end-of-program NOCTI assessment. One strategy to help CTE 
students achieve academically is integration of academics in CTE instruction. The 
Pennsylvania Department of Education Bureau of Career and Technical Education (PDE 
BCTE) implemented several professional development activities for teachers to increase 
academic achievement of CTE students. PDE adopted programs such as Governor’s 
Institutes where teachers and administrators of CTCs could attend and learn strategies to 
incorporate academics into all CTC programs (Pennsylvania Staff Development Council 
website, 2009). Currently there is another initiative called the Technical Assistance 
Program (TAP) also through Pennsylvania Department of Education (Bureau of Career 
and Technical Education website, n.d.). This is a mandated initiative for CTC schools in 
Pennsylvania in which CTC teachers, academic teachers, and CTC administrators learn 
how to integrate numeracy and literacy strategies into CTE disciplines. By relating to the 
sending districts that this suburban CTC could do more to help students meet proficiency 
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scores in the PSSAs, they may continue to send 10 th graders. Some part-time CTCs 
implemented additional strategies to remediate students. For example some schools, 
implemented competency-based electronic tutorials such as Study Island.” (Study Island 
website, 2010). This suburban CTC worked in concert with a sending district to let the 
CTC implement the electronic tutorial, Study Island, on site so the students would not 
have to be pulled back to the sending district for remediation. Study Island is a computer 
aided instruction software that is aligned with the PSSA achievement test required for all 
1 1 th grade students in Pennsylvania. This software, the focus of this study, will be used 
as an independent variable to determine if, when used, will help students learn and 
improve scores. 

The Problem 

This study will attempt to investigate if part-time CTCs can enhance students’ 
scores, at least to the proficiency level, in the PSSA state standardized assessment, the 
NOCTI assessment, and improve students’ overall educational experience by integrating 
academic assistance in grades 10 through 12. More specifically, the purpose of this study 
is to determine if Study Island, a Software package designed around the Pennsylvania 
State System of Assessment (PSSA), will aid IEP and non IEP students in achieving 
proficiency scores on both the PSSA and NOCTI exams in the part-time career and 
technical center of this study. The purpose of including IEP students in this research is 
because they make up 53 percent of the senior class that is the focus of this study. Unless 
the IEP students qualify for the Pennsylvania Alternate State Assessment, (PASA) they 
are part of the school’s overall accountability system. Thus, it is necessary to make sure 
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IEP students receive the necessary accommodations and assistance that will make them 
successful as well. Federal regulation, specifically United States Code 1412 (16), 
requires that students with disabilities must be included in state accountability systems 
and must be included in the assessment results reporting process (Cornell University Law 
School website, n.d). 

Overview of CTC of this study. 

The CTC of this study is a half-time technical school residing in suburban 
Philadelphia (hereafter referred to as Suburban CTC) for grades 10 through 12. With a 
few exceptions, the sophomores attend in the morning session and the juniors and seniors 
attend in the afternoon session. There is a 53 percent IEP population that is 
predominantly learning disabled. There are six cluster areas which include Architecture 
and Construction, Arts, A/V Technology and Communication, Human Services, 
Hospitality and Tourism, Manufacturing, and Transportation and Distribution. There are 
four major sending districts and one parochial school that feed this CTC. The male 
population makes up 68.5 percent of the total student population and the female 
population makes up 31.5 percent. 

Research Questions 

The following research questions will be addressed: 

1. To what degree does Study Island Software aid all students in reaching 

proficiency levels in the 4Sight Math and Reading Assessments in this part-time 



CTC? 
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2. To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in the 4Sight Math and Reading Assessments in this part-time 
CTC? 

3. To what degree does Study Island Software aid all students in reaching 
proficiency levels in state academic assessments (PSSA) in the part-time CTC of 
this study? 

4. To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in state academic assessments (PSSA) in the part-time CTC of 
this study? 

5. To what degree does Study Island Software aid all students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in 
the part-time CTC of this study? 

6. To what degree does the Study Island Software aid IEP students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in 
the part-time CTC of this study? 

Significance of the Study 

The significance of this study will help to determine if Study Island, a Software 
package developed in line with Pennsylvania Standards and common core indicators, will 
aid non-IEP and IEP students in achieving proficiency scores on both the PSSA and 
NOCTI exams in this part-time suburban CTC. Many part-time CTC schools have used a 
multitude of strategies and initiatives to help improve student academic learning for 
purposes of boosting career skills as well as improving academic achievement. Currently 
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there is not, nor has there ever been a part-time CTE in the state of Pennsylvania that 
made the threshold for AYP. 



Limitations 

There are a few limitations to this study including the diverse learning population, 
lack of parity of the different assessments, the frequency and amount of time each student 
spent on the Study Island Software, and the diverse academic backgrounds of the Study 
Island facilitators. The researcher recognizes that there are many factors other than Study 
Island that influence student achievement at the suburban CTC. The author is not able to 
isolate the effects of Study Island from other factors. Therefore, the methodology will be 
limited to the use of regression analysis that shows the relationship between student 
performance on Study Island and student achievement measured by standardized tests. 

First, the suburban CTC that was studied has a 53% IEP population including 
students who have been diagnosed with learning support, emotional support, gifted, 
economically disadvantaged, and students who have a 504 plan. A 504 plan is for 
students not qualifying for an individual education plan (IEP) and is aligned with the 
Americans with Disabilities Amendment Act (The ADA Amendments Act of 2008, n.d.) 
to accommodate students to be part of inclusion classrooms (Ed.gov: U.S. Department of 
Education, n.d.). In a larger academic facility, some of these classifications equal forty or 
more students which then can be categorized as groups that can be evaluated within their 
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own sub-groups. The school that was being studied is not large enough to have sub- 
groups this large. 

Second, the three assessments, Study Island, 4Sight, and PSSA exams, all have 
their way of scoring which is different from one another. Study Island exams, 4Sight 
exams, and NOCTI exams have purely raw scores whereas the PSSA state assessment 
reports raw scores and scaled scores. Even though the three tests have raw scores, the 
number of questions is different and a proficiency score in one is unrelated or unequal to 
a proficiency score in another. When comparing across tests, it is necessary to use cut 
score means and medians for proficient and advanced score interpretations. 

Finally, the original time-Table for Study Island work shops were originally 
designed for one half hour per week. Some students found that this was not enough time 
so extended their time to forty-five minutes or even an hour. Some students wanted only 
the one-half hour as they didn’t always take it seriously and just wanted it over with. 
Students were assigned user names and passwords and since this was a web-based 
program, students could log on at home as well. Some students did this and some did 
not. Although hours were tracked for time online, it didn’t necessarily mean it was 
productive time for the students. 

Definition of Terms 

Academic Anchors: Academic anchors stemmed from the academic standards and are 
specific to questions that may appear on the PSSA exams. (Pennsylvania Department of 
Education website, n.d.). Anchors are critical to CTE teachers in that they develop 
curriculum maps (Stone, III, Alfeld, & Pearson, 2008, p. 778) where they list each duty 




10 



and task and beside each task associate a corresponding math, reading, or science anchor. 
This helps the CTE teacher apply and incorporate the appropriate academic component 
relevant to the lesson. 

Eligible Content: Eligible content is actually the content of the standards that are 
examples of problems and questions in the form that they may appear on the actual exam 
(Pennsylvania Department of Education website, n.d.). This helps both academic and 
CTE teachers develop sample problems that are much like the eligible problems found on 
the 4Sight and PSSA Exams. 

Emotional Support : Term used to describe a diagnosable mental, behavioral or emotional 
disorder that lasts for a significant duration that meets the criteria within the Diagnostic 
and Statistical Manual of Mental Disorders (Pennsylvania Department of Education 
website, n.d.). 

IEP: Individual Education Plan 

Learning Disabled: Special education term used to define a disorder in one or more of 
the basic psychological processes involved in understanding or using language spoken or 
written that may manifest itself in an imperfect ability to listen, think, speak, read, write, 
spell or do mathematical equations (Understanding Special Education website, 2009). 
4Sight Exams: An exam created by PDE that contains a number of sample questions 
relating to the PSSA exams (Pennsylvania Department of Education website, n.d.). 

These were designed by PDE to help remediate students and prepare them for the PSSAs 
by using drill and practice theoretical framework. Schools that come under corrective 
action for not making AYP are required to purchase these and use these to help find 
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weaknesses and target those weaknesses in order to be successful on the state 
assessments. Schools use these however to mimic the PSSA Exams as they have the feel 
of the PSSAs. They can use these exams to target weaknesses and also to simulate a real 
time PSSA testing experience. 4Sight Exams are typically given 4 times a year in order 
to create benchmarks for improvement. Typically schools use an accompanying drill and 
practice Software to remediate and target weaknesses that emerge for each student in the 
4Sight Exams. 

504 Plan: Section 504 of the American with Disabilities Act guarantees that a child with 
a disability has equal ACCESS to an education and that it is comparable to an education 
provided to those who do not have a disability (Understanding Special Education 
website, 2009). 

Assumptions 

In this study, the following was assumed: 

1. Most students performed at a level to their best ability. 

2. The work done was the students’ own. 

3. Some students would not take the practice Software seriously. 

4. IEP students would not perform at the level of all students. 

5. The students would spend at least one half hour a week in school on the 
Software and more time at home for homework. 

6. There are many other initiatives besides Study Island that the CTC in this 
study was involved with to help influence state standardized tests such as: 



a. Governor’s Institutes 
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b. PA TAP Initiative 

c. MAX Teaching 

d. Professional Development relating to numeracy and literacy 

e. 4Sight testing that helped target weak areas 

f. More emphasis applied contextual learning 



Theoretical Framework 

Given the evaluation of high IEP population and low school performance, the 
theoretical model of this study focuses on remediation using strategies to improve both 
academic and CTE initiatives. The 4Sight assessment is used as a diagnostic tool to 
determine the academic levels of non-IEP and IEP students. There are many treatments 
used here but the ones used at this suburban CTC are classical conditioning by way of 
using computer aided instruction such as Study Island and applied and contextual 
learning for help with the NOCTI assessment. 

The first theoretical framework for this study is Classical Conditioning and the 
vehicle used to work within this framework is the computer aided instruction (CAI) 
practice. Classical conditioning is defined as, “A basic form of learning, sometimes 
referred to as Pavlovian Conditioning, in which a neutral stimulus is repeatedly paired 
with other stimulus that naturally elicits a certain desired response” ("Dictionary of 



Psychology," n.d.). Although CAI is used primarily to describe drill and practice 
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instruction, this study examines the ability of a software-based program to facilitate 
learning of new math and reading concepts in a variety of ways. In as far as student 
achievement and attitude towards learning, “research has not proven the efficacy of 
computers and software alone. However when used as a supplement to conventional 
instruction several researchers have found the results to be overwhelmingly positive, 
especially for learning disabled students” (Rosenberg, 2003, p. 1). The beneficial 
characteristics of CAI for students are immediate feedback, where students can ascertain 
their own weaknesses, receive immediate rewards when the correct answer is derived, 
can work at their own pace, and the computer grades without bias. Included in this 
theoretical framework for this study is the use of drill and practice (rote learning) to give 
students the necessary foundations to be able to go on and apply this fundamental 
knowledge to a higher level of critical thinking and problem solving. Rote learning is 
defined as, “Learning in a mechanical fashion through repetition, e.g. memorization, 
practice drills” (Herod, 2002, p. 1). Sometimes it is necessary to memorize quickly 
through drill and practice those fundamentals that allow one to perform higher level 
conceptualization and tasks so that they don’t get bogged down in minor details that 
would otherwise prevent you from solving a problem. CAI learning reflects behaviorist 
learning theory such as classical conditioning. For these students, lessons are chunked, 
students should receive constant reinforcement, lessons are pulled out of context, taught, 
and reintroduced back into context, and are teacher-centered. 



(See Figure 1) 
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The second theoretical framework is that of applied and contextual learning. 
Applied and contextual learning occurs when students can process new information or 
knowledge in such a way that it makes sense to them in their own frame of reference 
(Center for Occupational Research and Development, n.d). CTE by nature creates this 
type of learning. This suburban CTC has added academic standards only to where they 
specifically apply to a certain task. Once the academic standard is understood in context, 
the next step is to isolate that academic standard and teach students other applications in a 
more academic context. This helps both with CTE and academic skills. Sauders & 
Prescott (1999) cite Kolb (1983) as stating that, educators are encouraged to choose and 
design learning environments that incorporate as many different forms of experience as 
possible - social, cultural, physical, and psychological - in working toward the desired 
learning outcomes. Sauders & Prescott (1999) suggest a few means of presentations that 
lends themselves well to contextual learning. Hands-on learning is considered traditional 
in CTE and laboratory experiences that create excellent environments for knowledge, 
discovery, critical, and creative thinking. Project-based assignments involve many steps 
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and can take a long time to complete. It can involve other benefits such as group learning 
and is often interdisciplinary. For example, students involved in a motorcycle project 
could include machinists, welders, electronic technicians, and auto mechanics. 

Contextual connections use application of academics and can be accomplished by text 
introduction, media technology such as DVDs and CDs, internet searching, and 
interviews of people currently working in the profession of interest. Applications of a 
task or a project aid in terms of communication and presentation of ideas through 
presentations to teacher and classmates, technical writing, homework assignments, 
creating models of projects, and using slide presentations. The whole workplace learning 
concept can help students shadow with current employers, develop a mentoring 
relationship with subject matter experts, cooperative education, and apprenticeships. 
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Chapter 2 

Review of Literature 

Introduction 

This review of related literature focuses on the purpose of this study, which is to 
determine if Study Island, a Software package designed around the Pennsylvania State 
Systems of Assessment (PSSA) state assessment, will aid all and IEP students in 
achieving proficiency scores on both the PSSA and the NOCTI exams in a suburban part- 
time career and technical center. The focus will also help to examine the federal and 
state legislation’s No Child Left Behind (NCLB) (No Child Left Behind Act of 2001 
(Public law 107-110), 2001) impact on CTE in general, and what initiatives and steps 
CTE has taken to help address the ramifications. The reactions and efforts of education 
reform begin at the federal level, then trickle down to the state, local, school district, and 
finally to the individual school levels. 

Ravitch, (2010) states that the leading reform initiatives in American education 
were accountability and choice. Standardized test scores became the main focus of 
schools and these tests became the yardstick to measure the quality of schools. This is 
when the NCLB Legislation was passed and signed into law in January of 2002. Little 
did career and technical educators know how much this legislation would impact what 
they would teach and how they would teach it. Models were developed for enhancing 
math and reading improvement across academics. The career and technical education 
sector as educators felt that students, including career and technical education students, 
do not have the academic skills necessary for today’s high-skill workplace or entrance 
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into postsecondary institutions (Stone. Alfeld, & Pearson, 2008). Professional 
development using these models skyrocketed at an immense expense to the state and 
local school districts in Pennsylvania and across the country. 

NCLB Impact on Career and Technical Education 

NCLB has four primary areas that have possible ramifications for Career and 
Technical Education (CTE) (Kymes, 2004). First, it requires highly trained teachers. 
Teachers that graduate college for teacher education must now be highly qualified in the 
subject areas that they will teach. If they are a special education teacher, in order to be 
highly qualified, they must pass the Praxis exam in at least one area of the four core 
academic subjects; English, Math, Science, or Social Studies. Many special education 
teachers are highly qualified in more than one academic area. Also, a teacher can qualify 
for National Board for Professional Teaching Standards (NBPTS) Certification, which 
also identifies a teacher as highly qualified. The ramification for this is that a CTE 
educator may never qualify under this current category of NCLB because a CTE teacher 
can be hired right out of business and industry and obtain a state teacher certification 
while teaching. 

The second category in NCLB that may impact CTCs is the state accountability 
component. Career and Technical Centers (CTCs) are not directly accountable for 
academics. Sending districts are responsible and may choose to hold students back from 
CTCs in order to meet state guidelines and benchmarks in NCLB. 

CTE Institutions, at least in some states, have well-defined accountability 
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standards. While technically exempt from NCLB requirements, CTCs in 
most states depend heavily upon common schools’ cooperation for 
recruiting students. In order to maintain this cooperation, CTE institutions 
additionally may have to assume responsibility for the academic growth of 
their standards in mathematics and science. (Kymes, 2004, p. 3) 

Third is the aspect of research. The academic curriculum of all subjects must be 
rooted in some sort of applied research to ensure meeting the standards. Career and 
Technical Education is involved heavily in the sciences, and aligns with standards in 
business and industry which is also mostly related to the world of science. This in fact 
may serve as a benefit to CTCs as the sending districts may want to award credit for a 
student’s CTC discipline in the area of science. 

Finally, parental choice plays a significant role in impacting CTE. Parents have 
an option of having their children bussed to a different school or district that has made 
Adequate Yearly Progress (AYP) and if they decide to do so the sending district and the 
CTC could potentially lose a student. 

Students with IEPs. 

“Because NCLB rewards school districts with incentive awards based on student 
perfonnance, special needs students often feel personally responsible for their district’s 
failure to receive such rewards” (Kymes, 2004, p. 4). Most CTCs in Pennsylvania range 
between 30 and 50 percent special needs populations. Therefore, the opportunity for a 
CTC to share making AYP with their sending districts could be rare. CTCs calculate the 
percentage of students that attend their schools making a proficient score or better on the 
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PSSAs. When non IEP students’ scores at this suburban CTC alone are measured, the 
school could make AYP. But since this school has such a large portion of IEP students, 
the total junior population fails to make the bench mark scores. Both groups, the sending 
districts and the CTCs, must work together to meet the challenge of making AYP 
regardless of the high IEP enrollment. 

Pressures of statewide assessments. 

“American educators feel anxiety about improving student achievement now 
more than ever. Under the NCLB Legislation, districts and schools are held accountable 
to share performance data about student achievement in the form of a district report card” 
(Starmack, 2007, p. 10). The anxiety really begins at the beginning of the year when the 
eleventh graders arrive and teachers and administrators feel some relief but the anxiety is 
really not over until sometime in July or August when the scores are actually reported. 
The relief at best however is only temporary as in September, the cycle starts all over 
again. Through the NCLB Act, the state has placed minimum cutoffs as far as the 
percentages of students needing to score proficiently on the PSSAs in making AYP 
which means that the opportunity to make AYP gets harder every time the state raises the 
threshold. Ideally, by 2014, all schools will have reached 100 percent in the amount of 
students who score at least proficiently on the PSSAs. Given this, it will be even more 
intense in the 2010/201 1 school year as the bar is raised for cut scores for PSSA 
proficiency and above. The schools that barely made it in 2009/2010 will struggle to 
reach the new benchmarks for 2010/2011. The benchmark cutoff percentage for reading 
and math in 2009/2010 was 56 and 63 percent respectively for math and reading, while in 
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2010/2011, the percentages increases to 67 and 72 respectively (Pennsylvania 
Department of Education: Academic Achievement Report 2009/2010, n.d.). 

In need of improvement or corrective action. 

Regardless of what you may call failing to make AYP, in need of improvement or 
corrective action, “when you have that label, it’s a little cloud over your head. It really 
clears the air to have it removed” (D’Orio, 2009, p. 37). This is from a school that has 
been in need of improvement and worked its way back into making AYP. In 
Pennsylvania, once schools have made the corrective action list, they have to make AYP 
for two years in a row to be removed from the list. A huge responsibility falls on the 
principal to turn the school in a different direction and stress and anxiety infuse the 
climate and culture to the point of forgetting all the good things that schools do. It seems 
to become what defines what a school is and what goals becomes priorities. 

Teaching to the test mentality. 

Educators become so engulfed with the accountability to the state that they forget 
the pressure and angst that are placed on the students. 

These are difficult times for educators who believe that learning is worth 
pursuing for its own sake and that the chief purpose of school is the 
nurturing of students as whole human beings. Higher test scores seem to 
be the order of the day. To accomplish this aim, administrators strain to 
meet political agendas, teachers respond by teaching to the test, and 
students in return react by cheating, taking “learning steroids” (legal and 
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illegal psycho stimulants), or just not caring in order to cope with the 
demands placed on them in schools. (Armstrong, 2006, p.7) 

Exploratory learning, wonderment, culture enrichment, and being excited about 
learning life skills have been put asunder to make AYP. The spontaneity of learning has 
given way to rigorous, uniform, sequential learning. Education has been in danger of 
adopting a cookie cutter mentality and NCLB seems to take away individual styles of 
learning. State officials from the Pennsylvania Bureau of Career and Technical 
Education have followed suit by encouraging use of study guides for the NOCTI Exam 
and the practice Pre- NOCTI assessments. These Pre-NOCTI assessments help teachers 
target weaknesses in the students’ efforts and the teachers are given reports in each 
category of the exam on strengths and weaknesses. These efforts help primarily in the 
written portion of the NOCTI exam while the performance component still relies heavily 
on the student learning the skills necessary to function as an intern in his or her own 
chosen field. Professionals from business and industry create both the written and 
performance components to make sure the exams align with industry standards. 

Major CTE Initiatives 

Governor’s Institutes, High Schools That Work (HSTW), Technical Centers That 
Work (TCTW), the Technical Assistance program (TAP), Motivation, Acquisition and 
Extension (MAX) Teaching, and literacy and numeracy programs have emerged and an 
educator in CTE can’t attend a professional development workshop that doesn’t involve 
one of these initiatives. These initiatives were all developed in the name of NCLB to 
help improve PSSA scores, NOCTI scores, and to help maintain student enrollment. 
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Governor’s institutes. 

Although Governor’s Institutes do not exist anymore, they were instrumental in 
aiding CTE teachers in the process of academic integration. 

The Pennsylvania Governor’s Institutes for educators are part of a series 
of Summer professional educator’s programs designed to ensure the 
creation of challenging learning environments in the commonwealth 
public, private, and non-public schools. Each of the institutes provides an 
intellectually challenging program of study that will enhance academic 
classrooms and thereby assist educators in improving their students’ 
perfonnance and building capacity among educators. The Governor’s 
institutes for educators are intensive week-long professional development 
opportunities available for educators during the summer. The programs 
are rich with opportunities to deepen subject area knowledge and real- 
world experiences that help educators make the link to the Pennsylvania 
academic standards, reading and math education as a priority for all, 
classroom assessments and technology. (Pennsylvania Staff Development 
Council website, 2009, p. 1) 

Some of the professional development workshops included literacy, numeracy, data 
collection and analysis, Science, Technology, Engineering, and Math (STEM), CTE work 
standards, early childhood literacy, English Language Learners (ELL) strategies, 
improving school climate, and focus on urban education. 
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Contextual and experiential learning. 

Contextual and experiential learning theories have been major components 

used in CTE. 

According to the contextual learning theory, learning occurs only when 
students (learners) process new information or knowledge in such a way 
that it makes sense to them in their own frames of reference (their own 
inner worlds of memory, experience, and responses). This approach in 
learning and teaching assumes that the mind naturally seeks meaning in 
context, that is, in relation to the person’s current environment, and that it 
does so by searching for relationships that make sense and appear useful. 
(Center for Occupational Research and Development, n.d.) 

In concert with contextual learning, experiential learning uses the premise that 
learning by doing maximizes students’ ability to grasp difficult concepts. 

When education is said to be experiential, it means that it is structured in 
a way that allows the learner to explore the phenomenon under study - to 
form a direct relationship with the subject matter - rather than merely 
reading about the phenomenon or encountering it indirectly. Experiential 
learning, then, requires that the learner plays an active role in the 
experience and that the experience is followed by reflection as a method 
for processing, understanding, and making sense of it. 



(Education.com website, n.d.) 
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One of the most often used contextual learning theory problems is that of engine 
sizing. When teaching engine size, it is necessary to use the formula for a cylinder. The 
terms used in a cylinder of an engine are the bore and the stroke that relates directly to the 
diameter and the height of a cylinder. The formula for the volume of a cylinder is: 

V - m 2 h _ where V = volume, r = radius, and h = height. In the contextual learning 



approach the formula would be CV = k 




* s where CV = cylinder volume, the 



bore is the diameter, and s is the stroke or height. The bore (diameter) is divided by two 
to obtain the radius. Therefore, if the bore (diameter) is 4 inches, and the stroke (height) 
is 4.25 inches, the size of the cylinder would be 58 cubic inches. If we wanted to further 
calculate the engine displacement, we would multiply this result by the number of 
cylinders. If it were a six cylinder engine, the engine displacement would be 348 (350 
nominally) cubic inches. For an eight cylinder engine, the displacement would be 464 
cubic inches. These were popular engines when fuel economy was not an issue. The 
concept in learning contextually here is that there is an assumption that students are 
applying math to what they are truly interested in or at least feel is important to their 
trade competency area. The next step of course in rounding out their whole academic and 
technical experience is to transfer this knowledge into the math that would help the 
students calculate volume for any type of cylinder. In this case, by understanding 
language and literacy, the students can associate the word bore with diameter and stroke 
with height. Stone, III et al., (2008) quotes Fuchs et al., (2003) as stating, “Unless 
students are taught the abstract principle behind what they are learning in context and 
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guided through other contextual examples to which it applies, it is unlikely that cognitive 
transfer will occur outside the classroom ( p. 772). In other words, knowing how to 
calculate the cylinder volume of a vehicle is of little use in transference if CTE educators 
do not include academic examples outside of the realm of the discipline that the students 
are studying. The 1991 report by the Secretary’s Commission on Achieving Necessary 
Skills (SCANS) includes the following statement: 

We believe, after examining the findings of cognitive science, that the 
most effective way of learning skills is “in context” placing learning 
objectives within a real environment rather than insisting that the students 
first learn in the abstract what they will be expected to apply. 

(Secretary’s Commission on Achieving Necessary Skills [SCANS], 1991, 
p. viii). 

Bems & Erikson, (2001) in their article on contextual learning, use the contextual 
learning definition from the study conducted by the U.S. Department of Education at the 
Ohio State University in partnership with Bowling Green State University: 

Contextual teaching and learning is a conception of teaching and 
learning that helps teachers relate subject matter content to real 
world situations; and motivates students to make connections 
between knowledge and its applications to their lives as family 
members, citizens, and workers and engage in the hard work that 



learning requires. 
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Estepp and Norton (2003) list Stripling and Roberts’ (2010) seven steps for their 
experiential model shown at a poster presentation in State College, Pennsylvania during 
the North American Colleges and Teachers of Agriculture (NACTA). 

1. Assess students’ prior knowledge 

2. Create a common introductory experience 

3. Communicate the importance of the new educational experience 

4. Use a contextual learning experience 

5. Provide a reflection experience 

6. Provide a generalizing experience 

7. Provide a culminating application experience 

“Implementation of this model should allow students to create transferable knowledge 
which will then become prior knowledge for new learning experiences” (p. 19). 
Contextual and experiential learning are two important strategies to teaching and learning 
at this suburban CTC and are incorporated in all programs at all levels. 

High schools that work. 

High Schools That Work (HSTW) is an initiative developed in the southern states 
by the “Southern Regional Educational Review Board (SREB), State Vocational 
Education Consortium, a partnership of SREB, its member states, their school systems, 
and school sites” (About High Schools That Work website, 1999, p. 1). It currently 
consists of 1200 school sites in 30 states. 

The program is based on the belief that most students can master complex 
academic and technical concepts if schools create an environment that 
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encourages students to make the effort to succeed. Member schools implement ten 
key practices for changing what is expected of the students, what they are taught, 
and how they are taught. (About High Schools That Work website, 1999, p. 1) 

The ten key practices incorporate the concepts of high expectations, strong programs of 
study geared toward college entrance, academic studies, CTE studies, work-based 
learning, teachers working together, students actively engaged, strong guidance, extra 
help, and a culture of continuous improvement. This program was originally designed 
for comprehensive CTCs throughout the southern U.S. The issue up north was that most 
CTCs use the one-half day schedules so academics are isolated from the CTCs. The 
SREB developed a program from HSTW called Tech Centers That Work (TCTW) and 
Pennsylvania adopted their program through the Pennsylvania Technical Assistance 
Program (TAP) to work with part-time CTCs. It is the mission of TCTW is to create a 
culture of expectations and to help students to continually improve. Some of the goals of 
TCTW toward this end are to increase scores in reading, math, and science on the 
National Assessment of Educational Progress exam (NAEP), increase the percentages of 
students who complete a CTC program and enter into a field in which they studied, 
increase graduation rates, develop policies and leadership initiatives that sustain a school 
improvement effort, increase the percentage of students who go on to postsecondary 
education without having to take remedial courses, and to increase the percentage of 
students who pass employers’ exams such as national licensure, state exams, and 
industrial credentials like the ASE certification. Workshops for TCTW were held in 
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State College, PA where literacy and numeracy strategies were taught to academic, 
technical, and administrative personnel. 

MAX teaching. 

Max Teaching is a reading and writing strategy program developed by Dr. Mark 
A. Forget (MAX Teaching With Reading and Writing website, 2006). MAX Teaching is 
a staff development opportunity for schools that are interested in improving reading, 
writing, and learning skills of all students from first to thirteenth grades. Schools can 
purchase materials and books from this organization and have representatives from MAX 
Teaching come to their schools and provide as much as two days of professional 
development. After the in-service when teachers had some practice implementing these 
strategies, a representative from MAX Teaching will return to the school following up 
with modeling these strategies with students in the room while the classroom teacher 
observes and learns. 

Data collection and analysis. 

Another AYP improvement strategy is teachers and administrators developing 
action plans from data collection and data analysis (Horn, 2010). A sample district goal: 
“During the 2009/2010 school year, all staff will incorporate eligible content in every 
common assessment for all core academic subject areas” (Horn 2010, p.14). A sample 
school wide goal: “The percentage of students in the targeted cohorts achieving 
proficiency on the 2010 PSSAs in math will increase by twenty percent for the 
economically disadvantaged and Hispanic cohorts and ten percent for the individualized 
education plans (IEP) and English language learners (ELL) cohorts” Horn (2010). The 
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thinking here is that there is an increasing awareness of children’s success as the scores 
increase and success perpetuates success. Having this awareness and being able to see 
marked improvement encourages teachers to continue with their efforts and set even 
higher goals. “Many educators harbor negative perceptions of data use because in the 
past the data have been incomprehensible, unhelpful, or used solely for compliance 
purposes, but using data to inform instructional and management decisions has been a 
characteristic of high-performing, high- achieving schools” (Laird, 2008, p. 34). 

Pennsylvania technical assistance program. 

PDE developed the Technical Assistance Program (TAP) to package initiatives 
and provide support for schools with low performing data. TAP, in partnering with Tech 
Centers that Work (TCTW), currently has 3 cohorts and 54 CTCs involved in developing 
rigorous academic and technical programs across the state of Pennsylvania, and the goal 
is to have all of the CTCs across the state participate and complete the two-year cycle 
(The Pennsylvania Department of Education website, 2010). Each school that 
participates in a cohort sends technical and academic program teachers and 
administrators to the workshops where they define and decipher appropriate strategies to 
infuse these disciplines. PDE also supplies a liaison from the state to work with 
individual schools to provide them with guidance and assistance. TAP offers 
professional development activities that encompass many numeracy and literacy 
strategies. Some of the literacy strategies include anticipation guides, admit slips, 
alphabet review, Cornell notes, exit slips, Jigsaws, mnemonics, hunt for treasure, and four 
comers activity. All of these activities tend to enhance student reading and writing skills. 
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Using standards. 

Using standards for improvement of curriculum and assessment is essential 
(Association for Supervision and Curriculum Development [ASCD], 2000). One should 
employ backward design, which is a technique by which teachers design lessons from 
assessment and the standards. Standards-based education is the charged role of the 
teachers and it is a changing process and not an event (Association for Supervision and 
Curriculum Development [ASCD], 2000). Teachers should design instruction with the 
end in mind. Teachers should also post and explain the standards to students in the way 
that they understand them and explain how they are accountable for them. 

Additional strategies and initiatives for CTCs. 

To supplement the strategies described, schools have developed other initiatives 
to improve their students’ scores in the state academic and technical assessments. Many 
CTCs are using Software to help remediate students in reading and math. The CTC of 
this study is providing students with 30 to 45 minutes per week removed from their 
technical studies in order to enhance their academic experiences both at the sending 
districts and in their technical area of study. Schools have also purchased, from the 
Pennsylvania Department of Education, 4Sight exams which mirror the PSSA 
Assessment. These are mini-PSSA tests that allow students to practice taking the PSSA 
exams and also help to target weaknesses. Typically, schools will facilitate these exams 
four times in a year to create benchmarks. The school that I am studying in my research 



has adopted the 4Sight exam. 
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Computer intervention programs. 

In yet another academic strategy, schools have purchased computer math and 
reading software. “As a result of the shift to integrate technology administrators may 
spend large amounts of money to purchase computers, hardware, peripheral devices, and 
software applications in an effort to obtain the most sophisticated items” Jones, Taylor, 
Smith, & Smith (2007, p. 18) quote from (Materials Technology, 1990). Jones et al. 
(2007) created a study to determine if participants using a math-based computer 
intervention program improved work-related mathematical skills of ninth grade students. 
Among the independent variables examined were student type (at risk or regular 
education) and gender. The dependent variables were pretests and posttests generated 
from a quasi-experimental design to establish as much control as allowable in an 
experiment where experimental control is extremely difficult. In determining their first 
hypothesis, they wanted to see if there was a significant difference between pretest and 
posttest scores of all ninth grade students, the study found a significant difference 
between the pretest scores and the posttest scores. This was an indication that the 
computer-based math intervention program was effective. No significant difference was 
found for student type on the pretest, however there was a significant difference in the 
mean posttest scores of the at-risk and regular education groups. The at-risk students had 
a higher mean score than the regular education students. It is also stated that, “at risk 
students may respond better to the computer-aided instruction than regular education 
students” (Jones et al., 2007, p. 27). Also this success can be attributed to differentiated 
instruction and more time spent with at risk students. 
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Benefits of Computer Aided Instruction. 

A study was performed to determine if CAI made a difference in 1 1 th grade 
students’ attitudes towards biology (Soyibo & Hudson, 2000, p. 195). There was a 
control group where the teacher used lecture and discussion, and an experimental group 
where the students also used lecture and discussion but added a CAI component. There 
was an instrument used to measure attitude towards biology at the beginning of the study 
and at the end. Initially the attitude of the 1 1 th graders towards biology in the control 
group was significantly better than the attitude towards biology of the experimental 
group. After the study was done, it was determined that the attitude of the students in the 
experimental group increased and was significantly better than those in the control group. 
Also, there was a pre and posttest biology assessment for the 1 1 th graders. The mean 
pretest biology test score of the control group was significantly higher than the mean of 
the experimental group pretest score (M = 97.46, M = 84.83 respectively). However, the 
mean test score of the biology posttest was significantly higher for the experimental 
group than it was for the control group (M = 92.77, M = 89.24 respectively) 

Mioduser, Tur-Kaspa, & Leitner (2000) performed a reading study with 46 five 
and six year old IEP students that were divided into three groups. The first group used 
printed materials and computer assisted instruction with a special reading program. The 
second group used printed material only with the special reading program and the third 
group, which was the control group, was just given the traditional IEP reading program. 
The results are shown in Table 1. 



Table 1. 
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Mean, SD, % Improvement and F-Values for the Three Groups in the Computer Assisted 
Instruction Reading Test 





Printed and 
computer 
(n=16) 


Printed without 
computer 
(n=15) 


Control group 
(n=15) 








Increase Mean 


Increase Mean 


Increase Mean 


F 


Between 




(sd) 


% 


(sd) 


% 


(sd) % 




Groups 


Letter 


5.75 


26.1 


2.93 


13.3 


0.50 2.3 


25.67 


1>2>3 


naming 


(2.67) 




(1.91) 




(0.94) 


** 




Word 


3.94 


32.8 


2.73 


22.8 


.36 3.0 


18.85 


1>2>3 


recognition 


(1.24) 




(1.94) 




(1.55) 


** 




Phonological 


28.31 


23.2 


17.07 


14.0 


5.86 4.8 


49.30 


1>2>3 


awareness 


(8.25) 




(4.82) 




(4.47) 


** 





Note. ** P value < .005 Mioduser, D., Tur-Kaspa, H., & Leitner, I. (2000). The learning value of 

computer-based instruction of early reading skills. Journal of Computer Assisted Learning , 7(16), 



54-63. 



All three groups showed improvements in reading but the improvement for the group that 
used computer assisted instruction was significantly higher than the print-only group and 
the control group (Mioduser et al., 2000). Mioduser et al., (2000) present a caveat in 
using computer assisted instruction: 

After several decades of educational implementation of computer 
technology, it is agreed that the technology by itself means only the 
necessary infrastructure upon which should be built robust pedagogical 
solutions to real learning problems. Notwithstanding, when the new 
technology, the web, irrupted to the educational scene, the old pattern 
prevailed once again. Transitional stages at which new technologies are 
assimilated by means of known models are a reasonable (and perhaps 
unavoidable) phenomenon, only if they lead to mindful reflection and 



building of sound pedagogical applications of the new possibilities 
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in response to the learners’ needs, (p. 61) 

Computer assisted instruction in and of itself cannot stand alone as its only pedagogical 
method, but a blend of this strategy along with well-thought out other teaching strategies 
is necessary. The computer is merely a tool which students and teachers can use to 
enhance the teaching and learning process and this is the approach that this suburban 
CTC takes with the Study Island Software. 

Study Island Software. 

The school that is the subject of this research adopted the Study Island Software 
to help remediate students in reading and math through instruction, testing, and drill and 
practice. The software allows for web-based instruction, practice, testing, and rigorous 
academic content that is fun and engaging, is researched based with proven results, user 
friendly anywhere because it is web-based, and relatively affordable (Study Island 
website, 2010). This software keeps a data base of all students’ scores and monitors time 
on task. All students registered have their own personal database of information so that 
teachers and administrators can track progress or lack thereof. This study uses this 
database as part of a comprehensive independent variable to measure progress against 
4Sight and PSSA testing. Study Island includes mini lesson plans for students to see 
examples of how to solve problems. (See Figure 2) 

There are options in this software to view questions as they would be found on 
each individual state’s standardized tests or students can play games while learning. For 
example, there is a hockey component in which every time you answer a question 
correctly you have an opportunity to shoot at a goal in a given amount of time. Some 
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students prefer this and others prefer the standard methods of answering questions. Since 
the program is web-based, the students can work on the Software at home and it is often 
assigned by teachers as homework assignments. The managing of students and their 
progress can log on to each student’s site and determine if the student was guessing, how 
much time was spent on each item, and how many items a student completed. Teachers 
can then use a rewards based system of their choosing to reinforce student activity on the 
program. 

This suburban technical school also initiated a math in CTE challenge event in 
which students log onto Study Island, go to the custom screen, and answer questions that 
are associated with PSSA math but have a technical component included to make it 
applicable to the students’ programs. A nice feature of Study Island provides a custom 
area in the software that allows teachers to add their own CTE math related problems. 
There are prizes and awards for those who answer the most questions correctly. For 
example, a sample problem in fractions might ask, “If a plumber cut 17 3 / i” of V 2 ” copper 
from a length of five feet, how much copper would be left?” In the culinary arts 
program, a sample problem in fractions might ask, “If a recipe calls for 2 2/3 cups of 
flour for four servings, how many cups of flour would be needed for six servings?” 
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Square Roots 

The square of a number is that number times itself. 

A 2 = a x a 

The inverse of squaring is finding the square root. 

Example 

What is a square root of 100? 

Solution 
10 x 10 = 100 
Therefore, 10 2 = 100. 

So, a square root of 100 is 10 . 

Every positive number has two square roots that are opposite in sign, and the square root of zero is zero. 

Example 

What are the square roots of 4? 

Solution 

2x2= 2 2 =4 

(-2) x (-2) = (-2) 2 = 4 

So, the square roots of 4 are 2 and - 2 . 



The radical symbol V denotes the principal square root, which is the non-negative number that 
squares to equal the number under the symbol. 



Examples 

v 4 = 2 
v 25 = 5 
— v 49 = -7 



Figure 2. A sample mini lesson with examples as it would appear on Study Island. 
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Chapter Summary 

Chadd & Drage (2006) cite Brand (2003) as stating, “CTE programs are a vital and 
necessary component of the high school curriculum. Evaluations of CTE Programs in 
schools and districts show CTE programs contribute to increased school attendance, reduced 
high school drop out rates, higher grades, and increased entry into post secondary 
education” (p.8 1). Given this evidence of success, CTE educators need to do everything 
possible to integrate academic subjects into their technical subject matter to not only sustain 
success, but to continue to make improvements. Chadd & Drage (2006) also state 
that “studies have also been done to show that contextual learning is more beneficial and 
effective for students in retaining information” (p.84). 

One problem with contextual learning however is that students may not be able to generalize 
this information that was learned as applied to a task in a CTE discipline to generic math 
problems found on the PSSA assessment (Stone, et al., 2008). This is the reason that CTE 
educators have to also teach the applied academics in the generic sense after they are learned 
in a contextual environment. In the school that was studied, the teacher teaches the 
academic content contextually and generically, then students continue to learn and practice 
on the Study Island Software. In effect, the Study Island Software exercises combined 
with contextual learning and what they learn in their regular academic studies, combine to 
provide a comprehensive and full academic experience that will help them to be successful 



in NOCTI and academic state assessments. 
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Chapter 3 
Methodology 

The Problem 

The purpose of this study was to determine if Study Island, a Software package 
designed around the PSSA state assessment, will aid non-IEP and IEP students in 
achieving proficiency scores on 4Sight, the Pennsylvania State Systems of Assessment 
(PSSA), and National Occupational Competency Testing Institute (NOCTI) assessments 
in a part-time career and technical center (CTC). The underpinning of this study focuses 
around three core premises: the impact that A Nation at Risk (1983) had on the U.S. and 
State Departments of Education, the ripple-effect impact on career and technical 
education (CTE) of No Child Left Behind (NCLB), and what career and technical 
education agencies and schools have done and are doing to ameliorate any negative 
influences impressed upon them as a result of national, state, and local legislation’s 
imposition on traditional academic education. 

Research Questions 

In order to gratify this research’s problem statement, six research questions are 
explored in depth: 

1. To what degree does Study Island Software aid all students in reaching 

proficiency levels in the 4Sight Math and Reading Assessments in this part-time 



CTC? 
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2. To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in the 4Sight Math and Reading Assessments in this part-time 
CTC? 

3. To what degree does Study Island Software aid all students in reaching 
proficiency levels in state academic assessments (PSSA) in the part-time CTC of 
this study? 

4. To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in state academic assessments (PSSA) in the part-time CTC of 
this study? 

5. To what degree does Study Island Software aid all students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in 
the part-time CTC of this study? 

6. To what degree does the Study Island Software aid IEP students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in 
the part-time CTC of this study? 

Measurement 

Population. 

The population of students at the career and technical school of this study is 
mostly Caucasian, living in a suburban community, and are from relatively middle class 
to wealthy areas. There is a small minority from the local parochial school that does not 
participate in the PSSA assessments but participate in the Terra Nova exams, so these 
students are exempt from this study. The class of 2011 was studied. In this group of 251 
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seniors, 172 (69 percent) are male and 79 (31 percent) are female. Students enroll in one 
of nine career clusters consisting of Transportation, Architecture and Construction, 
Human Services, Hospitality, Health Science, Information Technology, Public Safety, 
Manufacturing, and Science, Technology, Engineering, and Math (STEM). 
Transportation, Architecture and Construction, Information Technology, Public Safety, 
and STEM are primarily male dominated disciplines at this school but non-traditional 
students are encouraged to enroll in any program they would like to pursue. Health 
Science, Hospitality, and Human Services are programs that have predominantly female 
enrollment. There are a total of 94 IEP students enrolled in the class of 2011; 79 are male 
and 15 are female. Seventy-three of the IEP students are learning support and 19 are 
emotional support. There is one student who is considered MR and one that is 
“unidentified” for IEP reporting purposes. 

Variables. 

Independent variables. 

One independent variable for this study is the initial score that the students 
receive on the Study Island Software as they log on, participate in the lessons, perform 
the drill and practice, and take formative assessments. These scores are recorded and 
posted on the Study Island Web Site where they can only be accessed by school officials 
that have the appropriate user name and password. 

The tenth grade students who participated in the Study Island Software 
remediation program are those students who did not perform to a proficiency level in the 
preliminary 4Sight Exam taken in September of their sophomore year. Those who did 
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perform at a Proficient level could opt out of this initiative. The students who need 
remediation are pulled from their technical programs approximately 30 to 45 minutes per 
week and can log on to computers in the resource center where monitors/instructional 
assistants (IAs) preside over the session. The IAs monitor student behavior and are there 
to offer assistance in any of the numeracy and literacy exercises that students may 
struggle with. Students are also able to log onto Study Island from home or a library 
computer because it is web-based software. The hours that they are logged on can be 
invigilated so that progress monitoring can occur. 

The software also has a way of determining if the students are merely guessing so 
that the overseer can determine if students are working through the process or merely 
using a process of elimination. In this way, if the teacher is using this Software exercise 
to evaluate student progress as part of a student’s grade, the teacher can speculate as to 
how much learning is actually occurring. 

The questions are generated randomly so the opportunity for a student to get a repeat 
question is extremely rare. This helps to determine if the students are learning the 
concepts and can transfer their learning to a multitude of problems. The Study Island 
Software also refers instructors and students to the applicable academic anchor from the 
Pennsylvania Academic Standards. (See Figure 3) Upon completion of a question or 
problem a student can access an explanation of the derived answer to reinforce their 
learning of the concept. (See Figure 4) In this case they explained that by converting the 
exponent to resemble the exponent of the first value it is then easier to perform a simple 



subtraction. 




42 



PA Grade 11, Math Anchor 
Mil. A. 1.1.1 

Find the square root of an integer to the nearest tenth using either a calculator or 
estimation. 

Covered by Study Island Topic: 

Square Roots 

Figure 3. Anchor sample from Study Island Software. 



(8.9 X 10 2 ) - (1.4 X 10‘ 4 ) 

First, convert (1.4 x 10" 4 ) so the exponent is the same as the first number, 

(8.9 X 10 2 ) - (0.014 X 10 2 ) 

Then, complete the subtraction. 

8.886 X 10 2 

Figure 4. Study Island explanation of correct answer. 



4Sight Assessment. 
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Another independent variable for this research is the scores students received in 
the 4Sight Assessment. 

The 4Sight Benchmark Assessments are typically quarterly benchmark 
assessments in reading and math developed for grades 3 to 11. Developed 
by the Success for All Foundation (SFAF) (Success for All Foundation, 
n.d.) with items field tested in Pennsylvania Schools, 4Sight mirrors the 
blue print of the PSSA and provides an estimate of student performance on 
the PSSA. 4Sight provides diagnostic information on Pennsylvania 
standards and specific sub-skills to guide classroom instruction and 
professional development efforts. (4Sight Benchmark Assessments, 2009, 

p. 2) 

Another independent variable is whether the student has or does not have an IEP. 
This is critical as this suburban CTC has a 53 percent IEP population and according to the 
Individuals with Disabilities Education Improvement ACT (IDEIA) of 2004, all IEP 
students must be measured and reported along with non-IEP students (Katsiyannis, 

Zhang, Ryan, & Jones, 2007, p. 161) 

Dependent Variables 

The dependent variables for this study were the PSSA and NOCTI scores. These 
scores will be compared to the class of 2010 scores to measure improvement if any. 
Caution must be taken here in that there is no claim that 4Sight Benchmark tests are a 
predictor of PSSA scores but only used as a measure of improvement from one quarterly 
assessment to the next. The CTC of this study only uses 4Sight twice yearly; once in 
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September and once in April. Time constraints of CTCs do not allow for more frequency 
than this. The test is time limited to 60 minutes and is meant to be on grade level for both 
literacy and numeracy. 

PSSA Assessments. 

The PSSA assessment is Pennsylvania’s state achievement test given to third, 
fifth, eighth, and eleventh grade students. It includes math, reading, writing, and science 
components. The PSSA Assessments are given in the Spring of each school year 
(window to be determined by the Pennsylvania Department of Education) for juniors. 
Even though reading, math, science, and writing assessments are all available and taken, 
CTC centers are primarily concerned with the math and reading components at this time. 
Students from some sending districts are excused from CTC attendance as the 
assessments are given all day. For other sending districts, attendance for the CTC is not 
impacted as the assessments are given during their regular academic class times. This 
suburban CTC attempts to hold their NOCTI assessments for seniors during the PSSA 
window so that juniors are not in attendance during the NOCTI exam. Results for the 
PSSAs are announced in July for sending districts. For those schools that administrate 
the PSSAs, they log on to a web site to access their scores. For CTCs, results come in the 
early Fall and can be accessed on the e-metric web site. (Data Interaction for 
Pennsylvania Student Assessments, n.d.) This is a very secure web site accessed only by 
educational administrative personnel after being given a PDE assigned username and 
password. This site gives both aggregated and disaggregated data to include race, gender, 
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special needs, and school district. It gives the raw scaled scores as well as the level of 
achievement: Advanced, Proficient, Basic, and Below Basic categories. 

NOCTI Assessment. 

The Carl D. Perkins Act of 2006 requires states receiving federal assistance for 
CTE to report a measurement of student technical skill performance. Pennsylvania 
implements a national test in each instructional program area. The tests are developed 
and scored by NOCTI. The NOCTI Assessment is given to seniors at the end of their 
program experience. 

Burke (1999) quotes Archibald and Newman (1988), “before educators try to 
assess authenticity, they should make sure they teach authenticity” (p.xvi). Career and 
technical educators teach authenticity as described by Burke (1999). Authentic educators 
use meaningful performance tasks, communicate clear industry standards and criteria for 
excellence, require quality work from students, emphasize meta-cognition and self- 
evaluation, teach skills that transfer, and create a positive interaction between assessor 
and student. The NOCTI exam accentuates all of these attributes of authentic learning by 
including both written and performance components in the exam. There is a performance 
piece and a written component. The written component consists of approximately 150 
questions, depending on the discipline, and related with it is a cut score developed by a 
team of professionals and craftsmen from each industry area. 

A student must score equally or better in both the written and performance 
portions of the exam to score at that level. The lower score determines the level of 
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achievement. If a student receives a Competent in the written and an Advanced in the 
performance, the student receives a Competent score on the NOCTI exam. 

Data Collection 

Scores from the Study Island, 4Sight, PSSA, and NOCTI assessments all come 
from secure web sites which are user name and password protected. The 4Sight reading 
and math assessments are given in September of the students’ sophomore year as a 
pretest to establish a baseline and in April as a posttest to measure improvement. Study 
Island tests are taken after each unit is completed. The PSSA assessment is taken in 
Spring of the students’ junior year and results aren’t usually available until the following 
Fall semester. The NOCTI assessment is given in the Spring of the students’ senior year 
and since it is the written portion is taken online, the scores are available almost 
immediately. The security of the PSSA and the NOCTI testing process is of the utmost 
importance and these tests are monitored. These results were tallied on a spread sheet 
and the students’ names and Pennsylvania Identification (PAID) numbers were coded so 
that confidentiality remained. Data were collected and transferred to a spread sheet and 
electronically stored on a computer which was username and password protected. The 
“honest broker method” was used to collect and analyze data i.e., the person at this 
suburban CTC, who was in charge of securing this data, randomly assigned numbers 
where names and PAID numbers were listed to ensure confidentiality and privacy that is 
essential to the Family Education Rights and Privacy Act (FERPA). 

The items that were stored on the spread sheet for Study Island included: 

(1) The number of items completed on the math section, 
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(2) The total score of all of the tests taken on the math section, 

(3) The number of items completed on the reading section, 

(4) The total score of all the tests taken on the reading section, 

(5) The total number of items completed for each of the subjects, 

(6) The combined score of the math and reading sections, 

(7) The level of competency the students scored on the math tests 
(Advanced, Proficient, Basic, or Below Basic), 

(8) The level of competency received on the reading tests, 

(9) Grade level at which students performed the exercises (eighth through 
eleventh), 

(10) The number of minutes students spent in each of the categories of 
math and reading. 

(11) For the 4Sight section the following categories will be listed in the 
spreadsheet: 

(a) The baseline math scores, 

(b) The baseline reading scores, 

(c) The final math scores, and 

(d) The final reading scores. 

(12) For the PSSA exams, 

(a) The math scaled score and 

(b) The reading scaled score will be inserted for both the 
2009/2010 and the 2010/2011 school year. 
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(13) For the NOCTI data, 

(a) The final written scores will be inserted for the 2009/2010 and 
the 2010/2011 school year. 

One important caveat to be aware of in this study is that NOCTI scores, although 
based out of a scale from 0 to 100, use cut scores to determine levels of success for each 
program and they are all different. As an example, the cut scores for the 2010/201 1 
NOCTI year for carpentry are 37.5 percent for Basic, 46.9% for Proficient, and 56.4% for 
an Advanced level for the written component of the NOCTI exam. 

The Nedelsky Method of determining cut scores is used by the team of experts. 
“The main premise of the Neldesky Method is that the test takers who do not know the 
correct answers to a question will eliminate as many answers as possible before making 
their final selection or guess” (Supernaw & Mehvar, 2002, p. 2). A substantial panel of 
qualified evaluators with clear instructions is used for this process. Each evaluator is 
instructed to review each question by crossing out the items that a minimally competent 
examinee should be able to eliminate. Each question is then given a reciprocal depending 
on the number of items remaining. The sum of the reciprocals over all items is denoted as 
the probable score of a minimally qualified examinee for a single evaluator. All evaluator 
results are averaged to arrive at the Competent Level. The Bureau of Career and 
Technical Education (BCTE) has determined the Advanced Level to be 2 standard errors 
of measure above the Competent Level and Basic Level to be 2 standard errors of 
measure below the Competent Level. 
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It is up to the panel of experts to determine which answers will be eliminated by 
the entry-level senior and proceed to cross those off. For example, in one question, one 
of the answers were eliminated as distracters, the three remaining answers would be 
reciprocated which would equal 1/3 or .67. If all of the questions for one evaluator 
equaled a mean score of .67, then that score would be averaged in with the other 
evaluators’ final scores. If all of the subject matter experts’ scores averaged to a .67, then 
the cut score for this exam would be 67 percent. Severe errors can be made here. If the 
subject matter experts assumed too much knowledge for the students, too many 
distracters would be taken away resulting in too high of a cut score. If the subject matter 
experts assumed no knowledge of a high school senior, then too low of a cut score could 
result. It can be very difficult then for an expert to think like a high school senior and 
errors in cut scores have resulted using this method. It is very critical that the Nedelsky 
Method is explained thoroughly and that each subject matter expert understands the 
process. Before the Spring of 2011, the performance part of the assessment had 
standardized cut scores in that an Advanced score is 80% or better, a Competent score is 
between a 75 percent and 79 percent, a Basic score is between 70 percent and 74 percent, 
and a Below Basic score is a sixty nine percent or lower. As of the 201 1 NOCTI testing 
date, the Angoff Method of establishing cut scores for the performance piece of the 
NOCTI was adopted (NOCTI Job Ready Criterion Referenced Cut-Score Project: 
Developmental Procedures, 2010, p. 1). In this method, subject matter experts are used 
to rate the difficulty of a task from a three to a five; five being the most competent and 
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three being minimally competent. All of the subject matter experts’ scores are averaged 
and changed to a percentage to arrive at the performance cut score. 

(14) The gender will be listed for each student as well as whether the 
student is a 

(15) IEP student and whether the 

(a) IEP student is learning support (LS) or is an 

(b) Emotional support (ES) student. 

Data Analysis 

Although formative and summative tests are easily accessible and do provide a 
level of valuable information about student progress, it is longitudinal data that tell a 
bigger story about long-term progress. “Educators often have access to various formative 
and summative assessment results, but leaders at all levels must demand, understand, and 
use longitudinal data to improve instruction and management” (Laird, 2008, p. 36). This 
in effect gives teachers more information such as information of increased or decreased 
improvement over the years. With this knowledge, teachers can tailor instruction to help 
students improve. As in the case of the Pre-NOCTI Assessments, teachers can zero in on 
sections of the written NOCTI that students were weak on and then can target these 
weaknesses for a better final NOCTI outcome. The longitudinal data provide valuable 
information to administrators that can help them manage and lead more effectively and 
with goals in mind such as percentage increases in the various assessments. Longitudinal 
data also empower administrators to calculate which initiatives show the best evidence of 
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increasing student achievement (Laird, 2008). The Pennsylvania Department of 
Education (PDE) has begun a longitudinal data system in many areas but for the purposes 
of Career and Technical Education (CTE) the e-metric site is the most appropriate. PDE 
began loading data to this site in 2002 but regarding PSSA information accessible to 
appropriate administrators only really began for the 2007/2008 school year (Data 
Interaction for Pennsylvania Student Assessments, n.d.). 

Both descriptive and inferential statistics will be used to determine gains in mean 
and median scores, correlations and regression analysis will be used to determine any 
relationship between Study Island and the three assessments, and regression analysis will 
be used to determine how much or to what extent the Study Island Software impacts test 
results. A Factorial ANOVA (sometimes referred to as a Factorial ANOVA) will be used 
to determine if any interaction of scores occur between Study Island and non-Study 
Island participants, and those with or without IEPs 

Table 2 lists the research questions, the key variables, and the data-analysis 
strategies that will be used in this study. The questions relate to how Study Island will 
aid IEP and all students’ success in state standardized academic and technical 
assessments in a part-time CTC. There will not be a total reliance on any one statistic but 
on a blend of the inferential data, particularly on regression analysis, correlation, and the 
p value, along with the descriptive statistics of the mean and median scores. The key 
variables of IEP, all, gender, Study Island scores, 4Sight pre and posttest scores, PSSA 
scores, and NOCTI scores are included. For our correlation results, Hinkle, Wiersma, & 



Jurs’ (1998) correlation rule of thumb was used. The Factorial ANOVA will also be used 
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to compare results of Study Island participation, whether the students have an IEP or not, 
and if there is an interaction between them. 

Table 2. 

Research Questions, Key Variables, and Data Analysis Strategies. 

Research Questions Key Variables Data Analysis-Strategies 



• To what degree does Study IEP students. All students. 

Island Software aid all students 
in reaching proficiency levels in Non IEP Students 
state academic assessments 

(4Sight &PSSA) in the part-time Study Island scores, 

CTC of this study? 

4Sight pre and posttest scores, 
PSSA scores, NOCTI Scores 



Descriptive Statistics (Mean, 
SD, Median scores, 
correlation) Inferential 
Statistics (regression analysis, 
R Squared Values, p values. 
Factorial ANOVA) 

Determine how much of the 
variances in PSSA and NOCTI 
is explained by Study Island. 



• To what degree does Study IEP students. All students. 

Island Software aid IEP students 
in reaching proficiency levels in Non IEP Students 
state academic assessments 

(4Sight &PSSA) in the part-time Study Island scores, 

CTC of this study? 

4Sight pre and posttest scores, 
PSSA scores, NOCTI Scores 



Descriptive Statistics (Mean, 
SD, Median scores, 
correlation) Inferential 
Statistics (regression analysis, 
R Squared Values, p values. 
Factorial ANOVA) 

Determine how much of the 
variances in PSSA and NOCTI 
is explained by Study Island. 



To what degree does the Study 
Island Software aid all students 
in reaching proficiency levels in 
the state end-of-program 
technical assessments (NOCTI) 
in the part-time CTC of this 
study? 



IEP students. All students. 

Non IEP Students 

Study Island scores, 

4Sight pre and posttest scores, 
PSSA scores, NOCTI Scores 



Descriptive Statistics (Mean, 
SD, Median scores, 
correlation) Inferential 
Statistics (regression analysis, 
R Squared Values, p values. 
Factorial ANOVA) 

Determine how much of the 
variances in PSSA and NOCTI 
is explained by Study Island. 
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Table 2 (Cont.) 



To what degree does the Study 
Island Software aid IEP students 
in reaching proficiency levels in 
the state end-of-program 
technical assessments (NOCTI) 
in the part-time CTC of this 
study? 



IEP students. All students. 

Non IEP Students 

Study Island scores, 

4Sight pre and posttest scores, 
PSSA scores, NOCTI Scores 



Descriptive Statistics (Mean, 
SD, Median scores, 
correlation) Inferential 
Statistics (regression analysis, 
R Squared Values, p values. 
Factorial ANOVA) 

Determine how much of the 
variances in PSSA and NOCTI 
is explained by Study Island. 



Chapter Summary 

This chapter focused on population, methods used for data collection and data 
analysis, dependent and independent variables, and a short description of all of the 
variables that will be used in this study. This will be a quantitative study using existing 
data from assessments that are used by the suburban CTC being studied which is stored 
on the various provider agencies’ web sites of the assessments being studied. The 
“honest broker” method will be used so that only the designated person at this school will 
know the names and Pennsylvania ID numbers of the seniors being studied. This person 
will use random numbers for identifiers so that the students will not be at risk of any 
confidentiality breaches. No local assessments will be used in this study but only the 
outside private and public tests and the Study Island computer assisted learning Software. 
This Software was originally designed for IEP students and filtered out to the all 
population. This is a nationally recognized and used Software and is targeted toward the 
individual state’s assessment systems. For Pennsylvania, this is the PSSAs which are 
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given to students in their junior year of high school. The 4Sight exams are typically used 
as a “corrective action” mandate for local education agencies (LEAs) and although not a 
computer assisted teaching and learning tool, it is used universally as an assessment tool 
that helps target weaknesses and guides educators and teachers to focus on those areas 
where students may need extra help. The extra help is facilitated through the Study 
Island Software package where students can view a lesson, practice their skills, and test 
out to monitor progress. 
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Chapter 4 
Analysis of Data 

The purpose of this study was to determine what impact the Software Study 
Island has on 4Sight, PSSA, and NOCTI assessments for a suburban part-time Career and 
Technical Center (CTC). The research method used was quantitative, and six research 
questions were used to help determine the outcome: 

1. To what degree does Study Island Software aid all students in reaching 
proficiency levels in the 4Sight Math and Reading Assessments in this part-time 
CTC? 

2. To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in the 4Sight Math and Reading Assessments in this part-time 
CTC? 

3. To what degree does Study Island Software aid all students in reaching 
proficiency levels in state academic assessments (PSSA) in the part-time CTC of 
this study? 

4. To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in state academic assessments (PSSA) in the part-time CTC of 
this study? 

5. To what degree does Study Island Software aid all students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in 



the part-time CTC of this study? 
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6. To what degree does the Study Island Software aid IEP students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in 
the part-time CTC of this study? 

Both descriptive and inferential statistics were used to compare the score means, 
standard deviations, maximum scores, and the analysis included correlation, Factorial 
ANOVA, and regression analysis. The senior class of 201 lwas followed from their 
sophomore year until their final NOCTI assessment in their senior year. All entering 
sophomores were given a 4Sight assessment in September of their sophomore year. 



MI 



•September of Sophomore Year 
•All students take 4Sight pretest 



H 

v : 



•Those who score below proficient 
•Participate in Study Island breakout sessions 



•April of Sophomore year 

•All students take 4Sight posttest 






M 



•September of junior year 
•All students take 4Slght pretest 



H 

v : 

\/ 



•Those who score below proficient 
•Participate in Study Island breakout sessions 

•Take PSSA or Keystone Exams 



Figure 5. Timeline from 4Sight pretest to PSSA assessment. 

Those that failed to score at a Proficient level or better participated in Study Island break- 
out sessions once a week for 30 to 45 minutes throughout the year. Those who scored at 



57 



a Proficient or Advanced level did not have to participate in the Study Island Software 
break-out sessions. All sophomores, however, took a 4Sight post-assessment in April. 
The same procedure was followed in their junior year but the Study Island sessions ended 
after the PSSA Assessment in April. All of these students then took a pre-NOCTI 
assessment in September of their senior year in preparation of the post-NOCTI 
assessment in March of 201 1. All of the existing data to include Study Island, 4Sight, the 
Pennsylvania System of State Assessments (PSSA), and NOCTI assessments were 
collected using the “honest broker” method. A person who secured all testing data in this 
suburban CTC listed all of the scores that each student received on the four assessments 
and randomly assigned numbers to each individual to ensure confidentiality and therefore 
none of these scores could be traced back to the individual specific name of the students. 

The study also included the IEP population scores to determine if a disability 
made a difference in 4Sight, the PSSA, and the NOCTI assessments while using the 
Study Island Software. Although the effects of gender were not examined in this study, 
Table 4 describes the gender breakdown of the senior class of this suburban CTC. 
Because the majority of the programs in this CTC related to male dominated occupations, 
the majority (68.59%) of the students are male. This represents about a two to one ratio 
of males to females. The females are mostly distributed among the traditional female 
occupations associated with enrollments in Cosmetology, Health Sciences and 
Occupations, Dental Assisting, and the Early Childcare Program. The Transportation and 
Construction Clusters are made up mostly of male students with a few non-traditional 



exceptions. 
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Table 3 shows the breakdown of gender in numbers and percentages. 



Table 3. 






Gender of Participants 






Gender 


n 


% 


Male 


172 


68.5 


Female 


79 


31.5 



Note, n = 251 

Table 4 shows the makeup of the IEP population by gender. Proportionately 
appropriate, the males make up most of the special IEP population. 



Table 4 
IEP Students 



Gender 


n 


% 


Males 


79 


84 


Females 


15 


16 



Note, n = 94 



Table 5 shows the breakdown of the students’ disabilities as they relate to learning 
support, emotional support, and other disabilities. The breakdown of specific IEP 
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classifications was not used but may be a topic for future study. The majority of the IEP 
students in this study relates to learning disabilities. 



Table 5. 

IEP by Disability 



Disability 


n 


% 


Learning Support 


73 


77.7 


Emotional Support 


19 


20.2 


Other 


2 


2 



Note, n = 94 



Research Question 1 - Proficiency in 4Sight math and reading for all students 

To what degree does Study Island Software aid all students in reaching 
proficiency levels in the 4Sight Math and Reading Assessments in this part-time CTC? 

The evidence that follows will show that for the 4Sight assessment, Study Island 
had negligible if any impact on the 4Sight reading and math tests. There was no 
improvement from the pretest to the posttest in reading which indicates that for whatever 
reason, students did not improve in the reading assessment. Study Island was not 
effective here. The non-Study Island participants scored higher than those who 
participated in Study Island. For math, those that did not participate in Study Island 
scored higher on the 4Sight math test than those who participated in Study Island. 
Although, there was a significant improvement in math from the pretest to the posttest, 
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the interactions between the conditions of whether students participated in Study Island 
math or not was not significant. 

Table 6 reflects the parameters used in the study to qualitatively describe 
correlation strength. These are general guidelines for Social Sciences and most of the 
correlations in this study fit into the moderate to low category. Table 7 reflects the 
parameters used in this study to determine the strength of the R“ effect. 



Table 6. 

Correlation Coefficients and Associated 
Strengths 



Strength of Correlation 


Very High Correlation 


±.90- 1.00 


High Correlation 


± .70 - ± .90 


Moderate Correlation 


± .50 - ± .70 


Low Correlation 


± .30 - ± .50 


Little if any Correlation 


.00 -± .30 



Note. Adapted from “ Applied statistics for the behavioral science ” Hinkle, D. E., Wiersma, 
W„ & Jurs, S. G. (1998). Boston, MA: Houghton Mifflin. (4th ed.) 

Table 7 

2 

R" Values and Associated Strengths 



Effects Size 


Small Effect 


0.01 - <0.09 


Medium Effect 


0.09 -<0.25 


Large Effect 


>0.25 



Note. Adapted from “ Statistical power for the behavioral 
sciences’’ Cohen, J. (1988). NY: Academic Press. (Revised ed.) 
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To follow the succession of how this class of students proceeded through the 
testing process, it is important to study the results of the 4Sight exams first, as they align 
with the PSSA assessments and are used as an improvement analysis tool for the PSSA. 

It is important to understand that the 4Sight process doesn’t assert to be a predictor of 
PSSA success, but only to be a practice tool to help students prepare themselves for this 
type of testing. 

Table 8 shows the results of regression analysis that was performed on the 
Study Island math and reading assessments with the post-4Sight assessment results. 

There was a positive correlation between scores in Study Island for reading and scores for 
the 4 Sight reading posttest (r = 0.517). This shows that the students who performed 
better on the Study Island reading also performed better in the 4 Sight reading. This was 
found to be significant, p < .05. The regression analysis showed an R 2 value of 26.8% 
which provides a measure of the shared variability and provides some evidence regarding 
the extent explained / predicted by Study Island scores in reading. 

There was a positive correlation between scores in Study Island for math and 
scores for the 4 Sight math posttest (r = 0.339). This shows that the students who 
performed better on the Study Island in math also performed better in the 4Sight in math. 
This was found to be significant, p < .05. The regression analysis showed an R 2 value of 
1 1.5% which provides a measure of the shared variability and provides some bases 
regarding the extent 4Sight reading tests can be explained / predicted by Study Island 



scores in reading. (See Table 8) 
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Table 8. 

Regression Analysis of Study Island with 4Sight Post Assessment Results by Subject: All 
Seniors 



Subject 


S 


R 2 


t 


P 


r 


Math 


13.43 


11.5% 


3.76 


0.000 


0.339 


Reading 


11.92 


26.8% 


6.57 


0.000 


0.517 



Note. S = standard error of the estimate r 2 = correlation squared, t = t test, 
p = < .05, r = Pearson Correlation 



A paired t-test was used to compare mean scores of the 4Sight pretest versus the 
scores of the 4Sight posttest for all seniors of the class of 201 1 sophomore year. Table 9 
reflects p values, the t values, and the confidence interval between the two tests for both 
math and reading. The reading portion of the 4Sight Assessment showed no difference in 
the pretest posttest mean scores at -0.10 (p = .927), while the math portion showed a 
mean difference of -3.94, (p < 0.001). The mean difference between the pre and post- 
reading 4Sight test was negligible. The 4Sight reading pretest had a mean of 58.39 
(SD=16.36) while the 4Sight reading posttest had a mean of 58.48 (SD= 18.54). A paired 
samples t-test showed no significant differences between the tests. For whatever reason, 
students did not demonstrate an improvement in reading throughout the year, at least 



when tested with the 4Sight. (See Table 9) 
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Table 9 



Paired t-test: 4Sight Reading and Math Pretest and Posttest. All Seniors 





n 


M 


SD 


95%CI 


t 


P 


Reading 














Pretest 


186 


58.39 


16.36 








Posttest 


186 


58.48 


18.54 








Difference 


186 


-0.09 


14.63 


-2.21,2.02 


-0.09 


0.927 


Math 














Pretest 


186 


43.74 


10.17 








Posttest 


186 


47.68 


12.26 








Difference 


186 


-3.94 


10.60 


-5.48, -2.41 


-5.07 


< 0.001 



The data in table 10 shows that non-Study Island participants averaged 70.85 
(SD=10.95) in their reading pretest and a 67.66 (SD=15.47) on their posttest, which 
means they did slightly worse on their posttest. For the students who participated in 
Study Island, their pretest mean score was 50.07 (SD=14.39), and their posttest was 
53.32 (SD= 18.18), which means they did slightly better on their posttest. 

When comparing the four conditions together it was shown that there was a 
significant difference between the pretest and posttest performance between the Study 
Island users and non-Study Island users, F=6.52, pc. 05. However, as the main effect 
between the pretest and posttest was negligible, it can be inferred that this value is 
significant because of how much better the non-Study Island user performed over the 
Study Island group. This statistic evidences that Study Island in itself had either no or a 
minimal effect on performance in the 4Sight reading test. 
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Table 10 

Examination of Study Island versus Non-Study Island Participants as 
measured for 4Sight Reading Pre and Posttests: All Seniors 

Variable n Mean SD F p 

Pretest 

SI 



No 


67 


70.85 


10.95 


Yes 


119 


50.87 


14.40 


Posttest 








SI 








No 


67 


67.66 


15.47 


Yes 


119 


53.32 


18.18 



Pre and posttest difference 6.56 .738 

Difference between SI and no SI 6.56 .01 1 

The data in Table 1 1 show that students who did not do Study Island scored a 
49.66 (SD=9.38) in their reading pretest and a 52.99 (SD=12.82) on their posttest, which 
means they did slightly better on their posttest. For the students who did participate in 
Study Island, their pretest score was 38.9 (SD=8.26) and their posttest was 43.25 (SD= 
10.18), which means they did slightly better on their posttest. 
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Table 11 

Examination of Study Island versus Non-Study Island Participants as 
measured for 4Sight Math Pre and Posttests: All Seniors 

Variable n Mean SD F p 

Pretest 

SI 



No 


76 


49.66 


9.38 


Yes 


110 


38.90 


8.26 


Posttest 








SI 








No 


76 


52.99 


12.81 


Yes 


110 


43.25 


10.18 



Pre and posttest difference 23.56 <.001 

Difference between SI and no SI .413 0.521 



The interaction between the conditions was not significant, which means that 
regardless of whether the students participated in Study Island or not, in they improved. 
This statistic evidences that Study Island had either no or a minimal effect on 
performance in math for the 4Sight test. 

Table 12 shows the regression analysis and correlation coefficients of all seniors 
relating the 4Sight posttest to the PSSA assessments taken by all seniors in the 2009-2010 
school years. Even though a modest mean difference exists between the 4Sight reading 
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assessment pretest and the 4Sight reading posttest, a moderate to high relationship exists 
in the correlation between the 4Sight posttest and the PSSA reading 
Table 12 

Regression Analysis PSSA Assessment versus 4Sight Posttest Results by Subject: All Seniors 



Subject 


S 


R 2 


t 


P 


r 


Math 


137.6 


46.4% 


12.7 


< .001 


.681 


Reading 


155.1 


49.2% 


15.09 


< .001 


.702 



Note. S = standard error of the estimate, R 2 = correlation squared, t = t test, 
p = < .05, r = Pearson Correlation 

portion of the analysis (r =.702). There is also evidence of a strong correlation in the 
math assessments with a R 2 value of 46.4 percent and a correlation coefficient of .681. 
The p values are both < .05 which shows a very strong statistical relationship. 

Research Question 2 

To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in the 4Sight Math and Reading Assessments in this part-time 
CTC? 

The evidence that follows will show that for the 4Sight assessment. Study 
Island had some impact on the 4Sight reading and math tests for students with IEPs. 

Even though the difference was not significant, the students without IEPs scored better on 
the 4Sight reading than students with IEPs. The non-Study Island participants scored 
higher than those who participated in Study Island. For math, Students without IEPs 
scored significantly higher on the math 4Sight than those with IEPs. Those that did not 
participate in Study Island scored higher on the 4Sight math test than those who 
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participated in Study Island. Although, there was a significant improvement in math 
from the pretest to the posttest, the interactions between the conditions of whether 
students participated in Study Island math or not was not significant. The correlation and 
regression data show considerable impact of Study Island of students with IEPs. 

The same regression and correlation analysis was performed for the IEP 
population for both math and reading. Table 13 shows the values of p, R , and 
correlation coefficients. Although the correlations are moderate, the R“ values are 
medium to strong per Cohen’s guidelines. There were significant improvements in the 
4Sight and PSSA scores for the IEP students, but the Study Island Software breakout 
sessions only account for 27.10 percent and 17.80 percent of the variances for math and 
reading scores respectively. 

Table 13 



Regression Analysis Study Island with 4Sight Assessment Results by Subject: IEP Students 



Subject 


S 


R 2 


t 


P 


r 


Math 


7.96 


27.10% 


4.35 


< .001 


0.520 


Reading 


16.79 


17.80% 


2.40 


0.020 


0.422 



Note. S = standard error of the estimate, R 2 = correlation squared, t = t test, 
P = < .05, r = Pearson Correlation 



There are many other initiatives that help IEP students achieve, and it’s a 
combination of all of these interventions including Study Island that help explain 
improvements for this population. However, significant positive relationships exist 
between Study Island and the 4Sight exams for both. 
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A paired t-test was done to compare mean scores of the 4Sight pretest versus the 
scores of the 4Sight posttest for sophomore IEP students for the class of 2011. Table 14 
reflects p values, the t values, and the confidence interval between the two tests for both 
math and reading. The reading portion of the 4Sight Assessment showed no significant 
difference in the mean scores (mean difference = 0.07, p = 0.965), while the math portion 
showed a significant mean difference (mean difference = 3.56, p = 0.003). (See Table 
14) 

Table 14 



Paired t-test: 4Sight Reading and Math Pretest and Posttest: IEP Students 





n 


M 


SD 


95%CI 


t 


P 


Reading 














Pretest 


86 


54.68 


16.82 








Posttest 


86 


54.61 


18.54 








Difference 


86 


0.07 


15.21 


3.18,3.33 


0.04 


0.965 


Math 














Pre -test 


86 


40.92 


9.41 








Posttest 


86 


44.48 


11.15 








Difference 


86 


-3.56 


10.78 


-5.87,-1.25 


-3.06 


0.003 



In reading, for students with IEPs, those who did not participate in Study Island 
scored a mean of 68.89 (SD=17.02) while those who did participate in Study Island 
scored a mean of 52.30 (SD=17.83). For students without IEPs, those who did not 
participate in Study Island scored a mean of 74.38 (SD=11.02) while those who did 
participate Study Island scored a mean of 55.071 (SD=17.25). There was no significant 
interaction between IEP’s and Study Island, which means that regardless whether the 
students did SI or didn’t do SI it didn’t necessarily have an effect on those students with 
IEPs or those students without IEPs, and vice versa. (See Table 15) 
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Table 15 

Examination of Study Island versus Non-Study Island Participants as 
measured for 4Sight Reading Posttests: IEP, Non IEP, SI and No SI 



Variable 


n 


Mean 


SD 


IEP 


86 


54.61 


18.54 


SI 


No 


12 


68.89 


17.02 


Yes 


74 


52.30 


17.83 


No IEP 


101 


61.76 


17.89 


SI 


No 


35 


74.38 


11.02 


Yes 


66 


55.07 


17.25 



Difference between IEP and Non IEP 
Difference between SI and No SI 
Interaction SENo SI, IEP/Non IEP 



1.79 


.183 


33.65 


< .001 


.193 


.661 



Since there was no significant interaction between whether a student 
had an IEP or participated in Study Island, Study Island did not necessarily 
have an effect on those students with or without IEPs. (See figure 6) 
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Estimated Marginal Means of FourSightFinalRead 




Figure 6. Interaction graph of Non-Study Island and Study Island participants 
and IEP and Non IEP students in 4Sight reading 

In the 4SIght math assessment, for students with IEPs, those who did not 
participate in Study Island scored a mean of 52.00 (SD=13.16) while those who did 
participate in Study Island scored a mean of 42.716 (SD=10.34). For students without 
IEPs, those who did not participate in Study Island scored a mean of 56.75 (SD=11.60) 
while those who participate in Study Island scored a mean of 46.41 (SD=11.51). There 
was no significant interaction between IEP’s and Study Island, which means that 
regardless whether the students did participate in Study Island or did not participate in 
Study Island, it didn’t necessarily have an effect on those students with IEPs or those 
students without IEPs, and vice versa. (See Table 16) 
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Table 16 



Examination of Study Island versus Non-Study Island Participants as 
measured for 4Sight Math Posttests: IEP, Non IEP, SI and No SI 



Variable 


n 


Mean 


SD 


F 


P 


IEP 


86 


44.01 


11.16 






SI 












No 


12 


52.00 


13.16 






Yes 


74 


42.72 


10.34 






No IEP 


101 


49.99 


12.50 






SI 












No 


35 


56.74 


11.60 






Yes 


66 


46.41 


11.51 






Difference of IEP and Non IEP 




4.05 


.046 


Difference between SI and No SI 




21.89 


<.001 


Interaction SI/No SI, IEP/Non IEP 




.063 


.803 



Since there was no significant interaction between whether a student had an IEP 
or participated in Study Island, Study Island did not necessarily have an effect on those 
students with or without IEPs and vice 



versa. (See figure 7) 
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Estimated Marginal Means of FourSightFinalMath 




Figure 7. Interaction graph of No Study Island and Study Island participants and IEP and 
Non IEP students in 4Sight math 

Research Question 3 

To what degree does Study Island Software aid all students in reaching proficiency 
levels in state academic assessments (PSSA) in the part-time CTC of this study? 

The evidence that follows will show that for the PSSA assessment, Study Island 
had negligible if any impact on the 4Sight reading and math assessments. The students 
who did not participate in Study Island scored significantly higher in the reading 
assessment that those who participated in Study Island. When regression and correlation 
analysis were used, the data shows that one may be able to use the Study Island scores to 
predict PSSA reading scores, but the data collected does not lend itself to determine 
whether Study Island had an effect on PSSA reading. In PSSA math, students who did 
not participate in Study Island scored significantly better than those who did not. The 
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2 

correlation and R“ values were to low and small to predict/explain any variation in PSSA 
math scores. 

There was a positive correlation between scores in SI for reading and scores for 
the PSSA reading (r = 0.359). This shows that the students who performed better on 
Study Island in reading also performed better on the PSSA in reading. This was found to 
be significant, p < 0.001. The regression analysis also showed an R" value of 12.9% which 
provides a measure of the extent to which differences in PSSA reading tests can be 
predicted / explained by Study Island scores in reading. (See Table 17) 

There was a positive correlation between scores in Study Island math scores with 
the PSSA math scores (r = 0.210). However, this correlation was not significant (p = 
.054). Therefore, there is no direct statistical relationship that can be inferred from how 
students perform in Study Island math and how they will perform in their PSSA math. 



Table 17 

Regression Analysis Study Island with PSSA Assessment Results by Subject: All Seniors 



Subject 


S 


R 2 


t 


P 


r 


Math 


180.7 


4.4% 


1.95 


0.054 


0.210 


Reading 


210.5 


12.9% 


3.75 


<.001 


0.359 



A two-sample independent t test was performed on the PSSA reading scores 
comparing the students who did not participate in Study Island (mean of 1344, SD=237) 



with the students who did participate in Study Island (mean of 1195, SD=224). There was 
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found a significant difference (t = 4.30, p <.001 ). It is important to note that there was a 
very large standard deviation within the scores. This shows that there was a significant 
degree of variability within the tests score. In other words, the students were all over the 
place. This statistic evidences that for whatever reason the students not selected for study 
island were able to perform much better on their PSSA in reading. (See Table 18) 

Table 18 

Two-Sample t test for PSSA Reading for those who participated and those who did not 
participate Study Island 



Participation 


n 


Mean 


SD 


SE Mean 


t P 


No SI 


83 


1344 


237 


26 


4.30 < .001 


SI 


97 


1195 


224 


23 





A two-sample t test was performed on the PSSA math scores for the students who 
did not participate in Study Island (mean of 1369,SD=217) with students who did 
participate in Study Island (mean of 1216, SD=184). There was a significant difference (t = 
4.94, p <.001). Another important statistic was that there was a very large standard 
deviation within the scores. This shows that there was a significant degree of variability 
within the tests score. This statistic indicates that for whatever reason the students not 
selected for Study Island were able to perform much better on their PSSA in math. (See 



Table 19) 
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Table 19 



Two-Sample t test for PSSA Math for those who participated in Study Island and who 
did participate use Study Island 



Participation 


n 


Mean 


SD 


SE Mean 


t P 


No SI 


84 


1369 


217 


24 


4.94 0.000 


SI 


85 


1216 


184 


20 





The PSSA with 4Sight posttest relationship results for all seniors by subject are 
shown in Table 20. The R" value for math was relatively moderate at 46.4 percent, and 
for reading it was 49.2 percent. However, the p values for both showed high significance 
at < .05 ip = 0.000 for both subjects). The correlations were .681 for math and .702 for 
reading which were both considered high. 

Table 20 



Regression Analysis PSSA Assessment with 4-Sight Posttest Results by Subject: 
All Seniors 



Subject 


S 


R 2 


t 


P 


r 


Math 


137.6 


46.4% 


12.7 


<.001 


.681 


Reading 


155.1 


49.2% 


15.09 


<.001 


.702 



Research Question 4 

To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in state academic assessments (PSSA) in the part-time CTC of this 



study? 
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The evidence that follows will show that for the PSSA assessment, Study Island 
had somewhat of an impact on the 4Sight reading and math tests for students with IEPs 
when analyzing regression and correlation. When compared with non IEP students 
however, the students with IEPs who participated in Study Island did not do as well. In 
terms of the reading, the students without IEPs scored higher than those with IEPs. The 
students who did not participate in Study Island scored higher in reading than those who 
did participate in Study Island. There was no significant interaction between whether a 
student had an IEP or not, or participated in Study Island or not, and vice versa. 

In the reading portion of the PSSAs for students with IEPs, those who did not 
participate in Study Island scored a mean of 1310 (SD=242) while those who did 
participate in Study Island scored a mean of 1144 (SD=210). For students without IEPs, 
those who did not participate Study Island scored a mean of 1422 (SD=226) while those 
who did participate in Study Island scored a mean of 1268 (SD=208). There was no 
significant interaction between IEP and Study Island, which means that regardless 
whether the students participated in Study Island or didn’t participate in Study Island 
didn’t have an effect on those students with IEPs or those students without IEPs, and vice 
versa. 

In terms of the PSSA reading scores, the difference between IEP students’ 

(M = 1180, SD = 227) and non IEP students’ scores (M = 1341, SD = 229) was 
significant in that students without IEPs scored higher (F= 10.64, p<.05). The 
difference between the scores for students who participated in Study Island and 
those who did not was also significant (F = 19.57, p < .05). Those who did not 
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participate in Study Island scored better in reading than those who did participate 
in Study Island. The interaction between Study Island participants and non- 
participants was insignificant. (F= 0.033, p >.05) (See Table 21) This means 
that whether a student participated in Study Island or not, or whether a student 
dad an IEP or not, had no impact on the outcome in the reading portion of the 
PSSA assessment. 

Table 21 

Examination of PSSA Reading and Study Island: IEP, Non IEP, SI and No SI 



Variable 


n 


Mean 


SD 


F p 


IEP 


87 


1180.83 


227.23 




SI 










No 


19 


1310.90 


242.74 




Yes 


68 


1144.49 


210.50 




No IEP 


93 


1341.44 


229.41 




SI 










No 


44 


1422.25 


226.34 




Yes 


49 


1268.88 


208.85 





Difference for IEP and Non IEP students 
Difference between SI and No SI 
Interaction SI/No SI, IEP/Non IEP 



10.64 


< .001 


19.57 


< .001 


.033 


0.857 
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As shown in figure 8, there is no significant interaction between IEP status and 
participation in Study Island which means that regardless if students participated in Study 
Island or not, it didn’t necessarily have an effect on students with IEPs and vice versa. 



Estimated Marginal Means of PSSARead 




Figure 8. Interaction graph of Non-Study Island and Study Island participant and 
IEP and Non IEP students 

In math for students with IEPs, those who did not participate in Study Island 
scored a mean of 1295 (SD=230) while those who did participate in Study Island scored a 
mean of 1158 (SD=171). For students without IEPs, those who did not participate in 
Study Island scored a mean of 1439 (SD=189) while those who did participate in Study 
Island scored a mean of 1312 (SD=182). There was no significant interaction between IEP 
and Study Island (F = .024, p = .576), which means that regardless whether the students 
did participate in Study Island or not didn’t necessarily have an effect on those students 



79 



with IEPs or those students without IEPs, and vice versa. 

In terms of the PSSA math, the difference between IEP students’ and non 
IEP students’ scores was significant in that students without IEPs scored higher 
(F= 22.58, pc. 05). The difference between the scores of students who 
participated in Study Island and who did not was also significant (F = 17.56, p < 
.05). (See Table 22) 

Table 22 

Examination of PSSA Math and Study Island: IEP, Non IEP, SI and No SI 



Variable 


n 


Mean 


SD 


IEP 


76 


1192.86 


195.49 


SI 


No 


19 


1295.21 


230.94 


Yes 


57 


1158.74 


171.17 


No IEP 


93 


1372.77 


195.44 


SI 


No 


63 


1439.52 


189.45 


Yes 


106 


1312.84 


182.50 



Difference for IEP and Non IEP students 
Difference between SI and No SI 
Interaction SI/No SI, IEP/Non IEP 



22.58 


< .001 


17.56 


< .001 


0.024 


0.576 
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As shown in figure 9, there is no significant interaction between IEP status and 
participation in Study Island which means that regardless if students participated in Study 
Island or not, it didn’t necessarily have an effect on students with IEPs and vice versa. 



Estimated Marginal Means of PSSAMath 




Figure 9. Interaction graph of Non-Study Island and Study Island participants and 
IEP and Non IEP students 

The PSSA and 4Sight posttest relationship results for IEP students by subject are shown in 
Table 23. The R" value for math was relatively high at 27.5 percent and for reading was 
32.9 percent. Also, the p values for both showed high significance at < .05. The 
correlations were moderate at .524 for math and .573 for reading. (See Table 23) 

Table 23 

Regression Analysis PSSA Assessment with 4Sight Posttest Results by Subject: IEP 
Students 



R 



Subject 



S 



t 



P 



r 
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Table 23 (Cont.) 



Math 


157.04 


27.5% 


9.07 


0.000 


.524 


Reading 


180.73 


32.9% 


11.89 


0.000 


.573 



Note. S = standard error of the estimate, R 2 = correlation squared, t = t test, 
r = Pearson Correlation 

A regression and correlation analysis of the PSSA reading and math assessments 
with the Study Island reading and math was performed. (See Table 24) In both subjects 
the p value was < .05 which indicates statistical significance. The R' value for reading 
was 49.2 percent which is a strong indication that Study Island explains about one-half of 
the variance in the PSSA reading assessment scores. The correlation for reading was .702 
which is high. The correlation value of .517 for math shows a moderate level, while the 
R 2 value was 26.8 percent. 

Table 24 

Regression Analysis PSSA Assessment with Study Island Assessment Results by Subject: IEP Ser 



Subject 


S 


R 2 


t 


r 


P 


Math 


145 


26.8% 


4.32 


.517 


0.000 


Reading 


157 


49.2% 


15.09 


.702 


0.000 



Note. S = standard error of the estimate, R 2 = correlation squared, t = t test, 
r = Pearson Correlation 

To take an even more detailed analysis at how the IEP students performed in the 
PSSA assessment, Table 18 shows descriptive statistics of the current IEP seniors versus 
the IEP seniors’ PSSA performance of the 2009/2010 school year. The mean score for 
reading for the previous year was 1140 versus a mean score of 1 158 for the current 
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seniors. This is a difference of +18 for the two scale scores. For math, the difference was 
+ 55. The IEP seniors of the class of 2010 were not involved in Study Island. Although 
the R" and the correlation values for reading were larger than those of math, the IEP 
seniors’ improvement gap from the previous year was larger for math than that of reading. 
(See Table 25) 



Table 25 

Descriptive Statistical Values of PSSA Years 09/10 - 10/11: IEP Students 



Subject 


Year 


n 


Mean 


Median 


SD 


Reading 


09/10 


101 


1140 


1155 


204.4 


Reading 


10/11 


119 


1158 


1166 


224.3 


Math 


09/10 


102 


1116 


1109 


181.1 


Math 


10/11 


104 


1171 


1172 


183.5 



Research Question 5 

To what degree does Study Island Software aid all seniors in reaching Proficiency 
levels in the state end-of-program technical assessment (NOCTI) in the part-time CTC of 
this study? 

The evidence that follows will show that for the NOCTI assessment, Study Island 
had negligible if any impact. Students’ scores increased significantly from the NOCTI 
pretest to the posttest. The students however, who did not participate in Study Island 



performed significantly better on the NOCTI posttest than those who did participate. 
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There was no significant interaction between pre and posttest NOCTI and whether 
students participated in Study Island or not. A correlation test showed no significant 
relationship and the R' value did not show that Study Island had an effect of NOCTI. 

Table 26 and Figure 10 show the number of students who scored at the Advanced, 
Competent, Basic, and Below Basic levels in the NOCTI pre and post assessments from 
2008 - 201 1. Each year the number of seniors who scored Advanced and Competent in 
the pre and post-assessments increased; while at the same time the numbers of students 
who scored Basic and Below Basic decreased. The largest decrease in the Below Basic 
level occurred from the 2009/2010 posttest to the 2010/ 201 1 post score tests. From 
2008/2009 to 2009/2010, the number of seniors who scored at the Below Basic level in 
the posttest increased by two points. These students had no exposure to Study Island. 
From 2009/2010 to 2010/2011 however, the number of students scoring at the Below 
Basic level decreased by 19 points; from 34 to 15. These students were exposed to Study 
Island. 

Table 26 

NOCTI Written Test Analysis 2008-201 1 





Advanced 


Competent 


Basic 


Below Basic 


2008-2009 Pretest 


77 


16 


53 


49 


2008-2009 Posttest 


98 


36 


47 


32 


2009-2010 Pretest 


83 


46 


55 


51 


2009-2010 Posttest 


112 


29 


31 


34 


2010-2011 Pretest 


103 


38 


36 


45 




84 



Table 26 (Cont.) 

2010-2011 Posttest 155 24 8 15 



Note. The pretest was taken in September and the posttest was taken in April of the same school year. 



Written NOCTI Scores 2008-2011 




■ Advanced 

■ Competent 

■ Basic 
Below Basic 



Figure 10. Number of seniors who scored at the Advanced, Proficient, Basic, and 

Below Basic levels in the pre and post NOCTI assessment from 2008 -2011. 

The number of Advanced scores rose from 98 to 112 from 2008/2009 - 2009/2010 
which is an increase of 14. The number of Advanced scores rose from 1 12 to 155 from 
2010/2011 which yields an increase of 43 seniors. Table 27 shows the percentages of 
seniors who scored at the four levels in the written, performance, and both the written 
Table 27 



Percentage of the Seniors scoring levels of Post- 
Seniors 


NOCTI Assessment, 


All 


NOCTI Component 


Advanced 


Competent 


Basic 


Below Basic 


Adv/Comp 


Written 


76 


12 


5 


7 


88 


Performance 


80 


13 


2 


5 


93 


Both 


62 


21 


5 


12 


83 
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and the performance levels. In order to receive a Governor’s Certificate, the students 
need to score at least a Competent level in both of the categories. Governor’s 
Certificates are awarded for those who score at the Advanced level on both the written 
and performance components. For those who receive at least a Competent score in one of 
the categories receive a certificate as well but is signed by the Secretary of Education 
rather than the Governor. Even though 88 percent scored at the combined 
Advanced/Competent levels in the written portion, and 93 percent scored at the combined 
Advanced/Competent levels in the performance portion, a total of 83 percent received a 
Competency or better for both. This means that some students received a Competency or 
better in one area but only a basic or below in another. 

A paired t test was performed on the NOCTI pre and posttests and the pretest 
had a mean of 61.87 (SD=13.54) while the posttest had a mean of 70.12 
(SD=12.73). This shows that the students’ performance increased significantly 
throughout the course of the year, at least on the NOTCI, t=l 1.95, pc. 05. (See 
Table 28) 

Table 28 

Paired t test for NOCTI Pre and Posttest 





n 


Mean 


SD 


SE Mean 


t 


P 


Post NOCTI 


156 


70.12 


12.73 


1.02 


11.95 


0.000 


Pre NOCTI 


156 


61.87 


13.55 


1.08 






Difference 


156 


8.26 


8.63 


0.69 
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Table 29 shows the between subject t test for the NOCTI pre and posttests. In 
both the pre and posttest, non-Study Island participants outscored Study Island 
participants and the difference was significant (p < .05). The mean score for the pretest 
for those who did not participate in Study Island was 64.96 while the mean score for the 
pretest for those who did participate in Study Island was 60.03. The mean score for the 
posttest for those who did not participate in Study Island was 73.81, while the mean score 
for the posttest for those who did participate in Study Island was 67.94. Both the 
difference between the pretest and the posttest was highly significant and the difference 
in scores of those who participated in Study Island and of those who did not was also 
highly significant. 



Table 29 

Examination for Study Island versus Non-Study Island Participants as measured for 
NOCTI Pre and Posttests: All Seniors 

Variable n Mean SD F p 

Pretest 

SI 



No 


58 


64.96 


13.50 


Yes 


98 


60.03 


13.31 


Posttest 








SI 








No 


58 


73.81 


12.04 


Yes 


98 


67.94 


12.68 
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Table 29 (Cont.) 






Pre and posttest difference 


7.17 


0.008 


Difference between SI and no SI 


7.17 


0.008 



Table 30 shows the regression and correlation data between the Study Island 
Math, Reading, and both subjects versus the post-NOCTI scores; the 4Sight Math, 
Reading, and both subjects versus post NOCTI; the PSSA Math, Reading, and both 
subjects versus the post NOCTI scores; and finally the pre-NOCTI versus the post- 
NOCTI scores. There exists no significant relationship with any of the Study Island 
assessments and the post-NOCTI scores. One could argue the significance of Study 
Island due to the fact that the only students who participated in the Study Island exercises 
are the ones that did not receive at least a Proficiency score on the baseline 4Sight exam. 
All students took both the baseline and final 4Sight exam so a better predictor of any 
influences would be one using the 4Sight assessments. It’s not until we reach the 4Sight 
reading assessment data do we begin to see a relationship, (p = 0.000, r = 0.359) The 
4Sight and the PSSA Reading components show a highly significant relationship (p = 
0.000), and a low to moderate correlation (r = .36 for 4Sight and a .43 for PSSA). 




Table 30 



Regression Analysis and Correlation of Study 
Post-NOCTI, All Seniors 

S R 2 

SI Math versus 12.85 0% 

Post NOCT1 

SI Read versus 12.42 2.9% 

Post NOCT1 

SI Both versus 12.63 1.2% 

NOCTI 

4Sight Math 12.24 2.5% 

versus Post 

NOCTI 

4Sight Read 11.67 12.2% 

versus Post 

NOCTI 

4Sight Both 11.71 12.2% 

versus Post- 

NOCTI 

PSSA Math 12.21 7.7% 

versus 

Post NOCTI 

PSSA Read versus 1 1.68 18.3% 

Post NOCTI 

Table 30 (Cont.) 



PSSA Both versus 
Post NOCTI 



11.64 



16.5% 



88 




5.69 



0.000 



0.427 
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Research Question 6 

To what degree does the Study Island Software aid IEP students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in the part- 
time CTC of this study? 

The evidence that follows will show that for the NOCTI assessment for students 
with IEPs, Study Island had negligible if any impact. Although there was not a 
significant difference in NOCTI scores for those students with IEPs and those who did 
not have an IEP, the students who did not participate in Study Island scored 
significantly higher than those who did. There was no significant interaction between 
whether a student participated in Study Island or not, or whether a student had an IEP or 
not and vice versa. 

Figure 1 1 shows the number of IEP seniors who scored at the Advanced, Competent, 
Basic, and Below Basic levels in the NOCTI Assessment. Fifty-nine students scored at 
the Advanced level, 11 scored at the Competent level, no students scored at the Basic 



level, and 7 students scored at the Below Basic level. 
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■ Advanced 

■ Competent 

■ Basic 
Below Basic 



Figure 11. n = 77. Number of IEP students who scored at different levels in the written 
portion of NOCTI. There were no Basic scores in this group. 



Figure 12 shows these numbers as percentages. Seventy-seven percent of the 



students scored at the Competent level or better in the written component of the exam, 



while only nine percent scored at the Below Basic level. 




Figure 12. Percentage of IEP students who scored at different levels in the written 
portion of NOCTI. There were no Basic scores in this group. 



For students with IEPs, those who did not do Study Island scored a mean of 72.79 
(SD=12.39) while those who did do Study Island scored a mean of 66.23 (SD=12.46). 
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For students without IEPs, those who did not do Study Island scored a mean of 73.87 
(SD=12.09) while those who did do Study Island scored a mean of 70.43 (SD=12.65). 
There was no significant interaction between IEP and Study Island, which means that 
regardless whether the students did Study Island or not didn’t necessarily have an effect 
on those students with IEPs or those students without IEPs, and vice versa. 

In terms of the NOCTI posttest, the students who did Study Island scored a mean 
of 67.91 (SD=12.64) while the students who did not scored a mean of 73.57 (SD=12.08), 
which means that the non-Study Island students performed better than the Study Island 
students on the NOCTI posttest, which proved to be a significant difference, F= 5.10, 
p<.05. (See Table 31) 



Table 31 

Examination of NOCTI test Study Island versus Non-Study Island Participants: IEP, 
Non IEP, SI and No SI 



Variable 


n 


Mean 


SD 


IEP 


76 


67.61 


12.66 


SI 


No 


16 


72.79 


12.39 


Yes 


60 


66.23 


12.46 


No IEP 


83 


72.21 


12.41 


SI 


No 


43 


73.87 


12.09 


Yes 


40 


70.43 


12.65 
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Table 31 (Cont.) 






IEP and Non IEP Difference 


1.42 


0.236 


Difference between SI and No SI 


5.11 


0.025 


Interaction SENo SI, IEP/Non IEP 


0.498 


0.481 



Figure 13 shows that there is no significant interaction between whether a 
student had an IEP or not and whether a student participated in Study Island or not. 



Estimated Marginal Means of NOTCTIPostTest 




Figure 13. Interaction graph of Study Island and Non-Study Island for NOCTI 
posttest 

Table 32 reveals some interesting data regarding PSSA and NOCTI. Of the 
students who scored Proficiency or better on both the reading and math components of 
the PSSA exams, 17 of those students scored a Competent or better on the NOCTI. This 
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is relatively a low number considering how well these students performed in the PSSA. 



Table 32 

Comparison of PSSA and NOCTI Levels for IEP Students 



Reading and Math 




NOCTI Competent 
Level 


NOCTI Advanced Level 


Proficient in Both 


17 


3 


14 


Non-Proficient in one area 


13 


3 


10 


.Non-proficient in both 
areas 


35 


6 


29 



Note. Proficient in PSSA is comparable to Competent in the NOCTI 



Of the students that scored a Proficient in at least one of the PSSA categories, 13 
students scored at the Competent and Advanced levels in NOCTI. Of the 35 students 
who did not score at least at the Proficiency level on either of the PSSA components, six 
received a Competent and 29 received an Advanced score. 

Table 33 shows regression Analysis and Correlation of Study Island, 4Sight, PSSA, 
and Pre-NOCTI versus Post-NOCTI for IEP Students. Unlike Table 29 that refers to all 
seniors, there is only one category that shows any significance; the PSSA Reading and the 
post-NOCTI. {p = 0.01 1, r = 0.303) for PSSA Reading. 
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Table 33 

Regression Analysis and Correlation of Study Island, 4Sight, PSSA, and Pre-NOCTI versus 



Post-NOCTI, IEP Students 





S 


R 2 


t 


P 


r 


SI Math versus 
Post NOCTI 


12.85 


1.5% 


0.86 


0.395 


0.124 


SI Read versus 
Post NOCTI 


12.30 


4.3% 


1.45 


0.154 


0.207 


SI Both versus 
NOCTI 


12.27 


4.7% 


1.70 


0.095 


0.218 


4Sight Math 
versus Post 
NOCTI 


12.14 


1.0% 


0.84 


0.402 


0.102 


4Sight Read 
versus Post 
NOCTI 


11.78 


6.7% 


2.21 


0.031 


0.259 


PSSA Math 
versus Post 
NOCTI; 


12.55 


4.7% 








PSSA Read 


11.68 


9.2% 


4.96 


0.093 


0.217 



versus Post 
NOCTI 

Table 34 shows the results of a paired t test was performed on the NOCTI pre and 
posttests and the pretest had a mean of 58.13 (SD=12.59) while the posttest had a mean 
of 67.72 (SD=12.71). This shows that the students’ performance increased significantly 
throughout the course of the year, at least on the NOTCI (t = -8.42, p<.05). 
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Table 34 



Paired T-Test and Cl: NOCTI Pretest, Posttest, IEP Students 





n 


M 


SD 


95% Cl 


T 


P 


Pretest 


75 


58.13 


12.59 








Posttest 


75 


67.72 


12.71 








Difference 


75 


-9.58 


9.86 


-11.85,-7.32 


-8.42 


0.000 



Note: Cl = Confidence Interval, p < .05 



Conclusions 

For the following analysis, the following was determined: 

• Study Island and 4Sight: All Seniors 

o The statistics used here evidence that although you can use SI reading 
scores as a predictor for how students are going to perform on the 4 Sight 
in reading, SI in itself does not actually have an effect on improving that 
performance for the 4 Sight test. 

o The statistics used here evidence that although you can use SI math scores 
as a predictor for how students are going to perform on the 4 Sight in 
math, SI in itself do not actually have an effect on improving that 
performance for the 4 Sight test. 

• Study Island and 4Sight: Seniors with IEPs 

o There was no significant interaction between IEP’s and SI, which means 
that regardless whether the students did SI or didn’t do SI, it didn’t 



necessarily have an effect on those students with IEPs or those students 




2 

without IEPs, and vice versa. However, there was a strong enough R' 
value to indicate that Study Island had some effect on 4Sight math. 

Study Island and PSSAs: All Seniors 

o The statistics used here evidence that although you can use SI reading 
scores as a predictor for how students are going to perform on the PSSA in 
reading, the data collected do not determine whether SI had an effect on 
the PSSA in reading. 

The statistics used here evidences that although you can use SI math scores as a 
predictor for how students are going to perform on the PSSA in math, the data 
collected does not lend itself to determine whether SI had an effect on the PSSA 
in reading. 

Study Island and PSSAs: Seniors with IEPs 

o There was no significant interaction between IEP’s and SI, which means 
that regardless whether the students did SI or didn’t do SI, it didn’t 
necessarily have an effect on those students with IEPs or those students 
without IEPs, and vice versa. However, regression analysis using Study 
Island and PSSA assessment shows a correlation and the R 2 values are 
strong enough to indicate that Study Island had some effect on PSSAs. 
However, there was a strong enough R" value to indicate that Study Island 
had some effect on PSSA math and reading. 

Study Island and NOCTI: All Seniors 
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o Although NOCTI scores improved immensely from the 2009/2010 school 
year to the 2010/201 1 school year, and improved immensely from the 
NOCTI pretest to the posttest, the statistics here used show no relationship 
with Study Island and NOCTI 
• Study Island and NOCTI: Seniors with IEPs 

o Even though the IEP students’ scores improved immensely from the 
2009/2010 to the 2010/2011 school year, and they improved immensely 
from the pre-NOCTI to the post-NOCTI, there was no significant 
interaction between IEP and SI for the NOCTI Exam, which means that 
regardless whether the students did SI or didn’t do SI it didn’t necessarily 
have an effect on those students with IEPs or those students without IEPs, 
and vice versa. 

Chapter Summary 

In the data analysis, an attempt was made to see what impact Study Island made 
on proficiency levels in the 4Sight, PSSA, and the NOCTI achievement tests. Descriptive 
statistics were used to determine if mean scores and median scores improved from the 
2009/2010 school year to the 2010/2011 school year. Inferential statistics were used to 
determine if the R" values, p values, and correlations were significant enough to 
determine if Study Island related positively to the 4Sight, PSSA, and NOCTI 
assessments. The Factorial ANOVA was used to see if there was an interaction between 
Study Island and non-Study Island participants, IEP and non-IEP participants, and 
between Study Island participants/non participants, and IEP/non IEP students. It was 
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found that for all students and IEP students, the single greatest factor in NOCTI 
improvement was the pre-NOCTI assessment. The pre-NOCTI assessment was designed 
to provide feedback to both teachers and students in which weak areas were identified to 
be targeted for improvement. 

Chapter five will discuss findings, implications, and recommendations for further 
study. It will explain why academic leaders should care about this study, which will 
benefit most from this study, and what CTC leaders, teachers, guidance counselors, and 
special needs coordinators can do with the results of this study. 




99 



Chapter 5 

The purpose of this study was to determine if Study Island, a Software package 
designed around the PSSA, will aid non IEP and IEP students in achieving proficiency 
scores on both the PSSA and NOCTI exams in the part-time career and technical center 
of this study. 

As superintendents, directors, and other key stakeholders who have an investment 
in curriculum look into purchasing remediation software such as Study Island, it is 
imperative that they consider the cost-benefit factor of such an initiative. For 4Sight 
testing, a renewal fee of $1000 annually is required to buy the site license. For the Study 
Island software, the cost is $1039 for each module purchased such as the eleventh grade 
math, eleventh grade reading, ninth and tenth grade PSSA math, and the ninth and tenth 
grade PSSA reading. See Table 35 



Table 35 


Annual Cost of 4Sight and Study Island Licenses 


4Sight 


$1000 


11 th Grade Math 


$1039 


1 1 th Grade Reading 


$1039 


9 th and 10 th Pre PSSA Math 


$1039 


9 th and 10 th Pre PSSA Math 


$1039 


Total 


$5156 



It might be well advised to fund initiatives that would better serve the Career and 



Technical Education curriculum. One such example might be to use funds to purchase 
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test-preparation study guides for NOCTI or other industry standard certifications that 
would enhance employability. Chapter’s four and what follows here in chapter five 
should be examined carefully before purchasing remediation software such as Study 
Island. 

Research Question 1 

To what degree does Study Island Software aid all students in reaching 

proficiency levels in the 4Sight Math and Reading Assessments in this part-time 

CTC? 

Findings 

Study Island and 4Sight Reading: All seniors 

It was determined by the data that Study Island had no impact on the 4Sight 
reading assessment. For whatever reason, the students made no improvement in reading 
as far as the 4Sight exam was concerned. 

A paired t-test was performed (Table 9) to determine if there were improvements 
from the 4Sight pretest to the 4Sight posttest for all seniors in reading. The test showed 
that for reading, the difference was negligible at -0.098 with a p value of .927 . This 
statistic shows that the seniors as a group made no improvements throughout the year in 
the reading portion of the 4Sight exam. These were all seniors so within that group, there 
were IEP students, non-IEP students, Study Island participators and non-Study Island 
participators. 

A Factorial ANOVA was then performed on this same group with students who 
participated in Study Island and with those who did not. With the pretest and the posttest 
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averaged together, the students who did not participate in Study Island performed better 
than those who did participate in Study Island. (F = 68.30, p < .05) A between-subject t- 
test was done to determine further how Study Island affected the 4Sight reading 
assessment and to see if there was an interaction. Table 10 in chapter four shows that the 
students who did not participate in Study Island went down from the pretest to the 
posttest, but still scored significantly higher in the pre and posttests than the Study Island 
participants. Table 10 also shows that the Study Island participators’ scores went up a 
little. (50.87 - 53.32) Since this is somewhat negligible, it can be determined that Study 
Island had no or a minimal effect on 4Sight exams. The p value for the pretest and the 
posttest was not significant at .738 while the p value for the differences in Study Island 
participants and non-participants was significant ( p = 0.01 1) in favor of those who did not 
participate in Study Island. 

Study Island and 4Sight Math: All Seniors 

Although the 4Sight math scores improved from the 4Sight pretest to the 4Sight 
posttest, it can be determined from the data that Study Island had negligible or no effect 
on this improvement. The students who did not participate in Study Island actually out 
performed those who did. 

A paired t-test was performed (Table 9) to determine if there were improvements 
from the 4Sight pretest to the 4Sight posttest for all seniors in Math. The test showed 
that, the difference was significant at -3.94 with a p value of 0.000. This statistic shows 
that the seniors as a group made improvements throughout the year in the math portion of 
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the 4Sight exam. These were all seniors so within that group, there were IEP students, 
non-IEP students, Study Island participators and non-Study Island participators. 

A Factorial ANOVA was then performed on this same group with students who 
participated in Study Island and with those who did not. With the pretest and the posttest 
averaged together, the students who did not participate in Study Island performed better 
than those who did participate in Study Island. (F = 63.63, p < .05) A between-subject t- 
test was done to determine further how Study Island affected the 4Sight math assessment 
and to see if there was an interaction. Table 1 1 in chapter four shows that students who 
did and did not participate in Study Island, went up in the posttest from the pretest. 
However, the non-Study Island participants scored significantly higher in the pre and 
posttests than the Study Island participants. This evidences that overall student 
performance in math increased throughout the year. The p value for the pretest and the 
posttest was significant at 0.000 while the p value for the differences in Study Island 
participants and non-participants was not significant, (p = 0.521) 

In the regression analysis of Study Island versus 4Sight, the correlation for each 
subject was a little different, (r =0.339 for math and r = 0.5 17 for reading) The p values 
were identical at 0.000 which shows a high significance in both subject areas. 

The major difference in this comparison was the R~ values reading was 26.8 
percent while 1 1.5 percent for math. The Study Island test is a good predictor of how 
students will perform in the 4Sight and the R“ value is strong enough to determine that 
Study Island explains the variance in the 4Sight exams for reading but not for math. (See 



Table 8) 
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Conclusions 

For the 4Sight reading test, we can determine that there was negligible or no 
improvement from the pretest to the posttest. However, for the 4Sight math, the data 
show that there was a significant improvement from the pretest to the posttest for 
whatever reason. As far as what effect Study Island may have had on the 4Sight 
outcomes, the data shows that Study Island could be used as a predictor of how well 
students will perform on the 4Sight, but Study Island only showed a strong effect on the 
reading portion of the 4Sight test. The data also show that the students who did not 
participate in Study Island outperformed those who did. 

Research Question 2 

To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in the 4Sight Math and Reading Assessments in this part-time CTC? 

It was determined by the data that Study Island had no impact on the 4Sight 
reading assessment for students with IEPs. For whatever reason, the students made no 
improvement in reading as far as the 4Sight exam was concerned. 

Findings 

Study Island versus 4Sight Read Posttest: IEP Students 

A paired t-test was performed (see Table 14) to determine if there were 
improvements from the 4Sight pretest to the 4Sight posttest for those students with an 
IEP in reading. The test showed that for reading, the difference was negligible at 0.07 
with a p value of .965. This statistic shows that the seniors with an IEP made no 
improvements throughout the year in the reading portion of the 4Sight exam. 
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In terms of the 4Sight reading posttest, the students with an IEP scored a mean of 
54.61 (SD = 18.54) while the students without an IEP scored a mean of 61.76 (SD 
=17.89) which means that the non-IEP students performed slightly better than the IEP 
students. However, the difference between the two populations was not significant, (p = 
0.183) As shown in Table 15, the difference however between those who did Study 
Island and those who did not was almost 20 points (F = 33.65, p = 0.000) with the non- 
Study Island group performing better. 

Study Island versus 4Sight Math Posttest: IEP Students 

The data shows that although the students without IEPs scored higher on the 
4Sight that those with IEPs, and the non-Study Island participants scored higher than 
those who did participate, the correlation between Study Island and the 4Sight test was 
moderate and the R 2 value was large which means Study Island was a good predictor of 
4Sight scores in math and that Study Island had an effect on math improvement for 
students with IEPs. 

A paired t-test was performed (see Table 14) to determine if there were 
improvements from the 4Sight pretest to the 4Sight posttest in math for those students 
with an IEP. The test showed that for math, the difference was significant at -3.56 with a 
p value of .003. This statistic shows that for whatever reason, the seniors with IEPs made 
significant improvements throughout the year in the math portion of the 4Sight exam. 

In terms of the 4Sight math posttest, the students with an IEP scored a mean of 
44.01 (SD = 11.16) while the students without an IEP scored a mean of 49.99 (SD = 
12.50) which means that the non-IEP students performed slightly better than the IEP 
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students. The difference between the two populations was significant, (p = 0.046) As 
shown in Table 16, the difference between those who did Study Island and those who did 
not was almost 1 1 points which was also found to be significant (F = 33.65, p = 0.000) 
with the non-Study Island group performing better. 

In the regression analysis of Study Island versus 4Sight for IEP students, the 
correlations for both subjects were a little different, (r =.520 for math and r = .422 for 
reading) The p values were a little different at 0.000 for math and 0.020 for reading but 
both still show a high significance. The R 2 value was 17.80 percent for reading which 
shows a medium effect and a 27.10 percent for math which was a large effect that Study 
Island had on 4Sight for IEP students. The data indicate that IEP statistics in regression 
analysis between Study Island and 4Sight were positive. (See Table 13) 

Conclusion 

There was a significant difference in the pre-4Sight test and the post-4sight test 
which shows that for whatever reason, the IEP students were able to improve in math at 
least in terms of the 4sight exam. The data show that non-IEP students outperformed the 
IEP students but the difference was barely significant which one could argue that the IEP 
students kept pace with the non-IEP students. Those IEP students who did not participate 
in Study Island also outscored those who did participate, and the difference was 
significant. The data show no relationship between those who were or were not IEP 
students, and those who did or did not participate in Study Island. The correlations show 
that you can use Study Island to predict how a student will perform on the 4Sight exam. 
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At least for reading, the data shows that Study Island had a strong effect on the 4Sight 
exam. 

Research Question 3 

To what degree does Study Island Software aid students in reaching proficiency 
levels in state academic assessments (PSSA) in this part-time CTC? 

Findings 

Study Island and Non-Study Island Participants and PSSA Read: All Seniors 

It was determined by the data that Study Island had little or no impact on the 
PSSA reading assessment. The students who did not participate in Study Island 
outscored those who did. 

Because there is no PSSA pretest, it is very difficult to determine to what extent 
Study Island affected the improvement in the reading assessment. Therefore, a between- 
subject t-test was performed. The students who did not participate in Study Island out 
performed those who did participate in Study Island. The difference was found to be 
significant. (See Table 18) 

In the regression analysis performed on the PSSA Assessment and Study Island, 
the correlation for Reading was low to moderate which shows that students who 
performed better on Study Island reading performed better on the PSSA. The p value 
showed a high significance (p <.05) and the R” value for reading was moderate at 12.9 
percent which shows that Study Island had a moderate effect on the PSSA. (See Table 



18) 
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Study Island and Non-Study Island Participants and PSSA Math: All Seniors 

It was determined by the data that Study Island had little or no impact on the 
PSSA math assessment. The students who did not participate in Study Island outscored 
those who did. 

Because there is no PSSA pretest, it is very difficult to determine to what extent 
Study Island affected the improvement in the Math assessment. Therefore, a between- 
subject t-test was performed. The students who did not participate in Study Island out 
performed those who did participate in Study Island. The difference was found to be 
significant. (See Table 19) 

In the regression analysis performed on the PSSA Assessment and Study Island, 
the correlation for math was low to moderate which shows that students who performed 
better on Study Island reading performed better on the PSSA. The p value showed a high 
significance (p <.05) and the R" value for math was low at 4.4 percent which shows that 
Study Island had minimal or no effect on the PSSA. (See Table 17) 

Conclusion 

The students who did not participate in Study Island out performed those students 
who did. The difference was significant in both math and reading. The regression 
analysis showed the R2 values of 12.9% and 4.4% in reading and math respectively, and 
although the correlation statistic provides a measure of the variability to which future 
PSSA reading tests can be predicted by SI scores, the data here indicate that Study Island 



had little or no effect on the PSSA assessments. 
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Research Question 4 

To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in state academic assessments (PSSA) in this part-time CTC? 

Findings 

PSSA Assessment and Study Island Read: IEP Seniors 

It was determined by the data that Study Island had some impact on the PSSA 
reading assessment for students with IEPs. The students with IEPs who did not 
participate in Study Island outscored those who did. The correlation and R“ values were 
moderate and large respectively, indicating that Study Island may be a good predictor for 
PSSA scores and that Study Island had some effect on PSSA outcomes. . 

The non-IEP students scored 192 points higher in the PSSA reading assessment 
than the IEP students. The difference between the groups was significant. For students 
with IEPs, those who did not do Study Island scored 166 points higher than those 
students with IEPs who did Study Island. 

PSSA Assessment and Study Island Math: IEP Seniors 

It was determined by the data that Study Island had some impact on the PSSA 
math assessment for students with IEPs. The students with IEPs who did not participate 
in Study Island outscored those who did and there was no interaction between Study 
Island and whether a student had an IEP or not. However, the correlation and R' values 
were strong and large respectively, indicating that Study Island may be a good predictor 
for PSSA scores and that Study Island had some effect on PSSA outcomes. 
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The non-IEP students scored 180 points higher in the PSSA math assessment then 
the IEP students. The difference between the groups was significant. (F = 17.56. p = < 
.05) For students with IEPs, those who did not do Study Island scored 137 points higher 
than those students with IEPs who did Study Island. As shown in figure 11, there is no 
significant interaction between IEP status and participation in Study Island which means 
that regardless if students participated in Study Island or not, it didn’t necessarily have an 
effect on students with IEPs and vice versa. 

Regression analysis PSSA Assessment versus Study Island Assessment, Math 
and Reading: IEP Students 

A regression analysis was done between PSSA and Study Island reading for 
students with IEPs. The correlation in reading was considered moderate at r = .573 with 
a strong R2 of 32.9 percent. The p value showed high significance at 0.000. Math was 
almost equally as strong, carrying a moderate correlation of r = .524, a p value of 0.000, 
and an R 2 value of 27.5 percent. (See Table 24) According to Cohen (1988) the R 2 
values in both cases are strong. 

Conclusion 

Non IEP students significantly outscored the IEP students in the PSSA 
assessments. The IEP students who did not participate in Study Island significantly 
outscored those who participated. There was no significant interaction between the two 
to show that one affected the other. The regression analysis data show however that 
Study Island can be used as a predictor to how students will do in the PSSAs and the 
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strong R" values and p values show that for the IEP population in general, Study Island 
had an effect. 

A comparison was done between the 2009/2010 school year and the 2010/201 1 
school year for PSSA Scores using descriptive methods. The mean score for reading in 
the 2009/2010 school year was (M = 1 140) while for the 2010/201 1 school year was (M 
= 1 158); an improvement of 18 points. The mean score for math in the 2009/2010 school 
year was (M = 1116) while the mean score for the 2010/201 1 school year was (M = 
1171); which was an improvement of 55 points. Although the gain was significantly 
greater in math, the reading scores were still higher for this class. (See Table 26) 

Research Question 5 

To what degree does Study Island Software aid students in reaching proficiency 
levels in the state end-of-program technical assessments (NOCTI) in this part-time CTC? 

Findings 

Study Island and NOCTI: All Seniors 

The data indicates that Study Island had little or no impact on NOCTI for all 
seniors. Students that did not participate in Study Island outscored those that did. The 
correlation and R“ values were small. 

A paired t-test was performed for the NOCTI pretest and posttest for the written 
portion of the assessment for all seniors. The mean score for the pretest was 61.87 (SD = 
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13.54) while the mean score for the posttest was 70.12 (SD = 12.73). A t value of 11.95 
and a p value of < .05 indicate a significant increase with a difference of 8.26. 

When a between subject t-test was performed for Study Island participants/non 
participants and the NOCTI posttest, it showed that for whatever reason, the students 
who did not participate in Study Island performed better on the posttest than did the 
students who participated in study Island. 

When a Factorial ANOVA was performed using the NOCTI pre and posttests’ 
differences, and whether Study Island was used or not, the differences between the pre 
and posttests for NOCTI were about the same. This means that it appears that Study 
Island had a minimal impact on the NOCTI assessment and even though the data does not 
indicate this, one might be able to argue that without Study Island, the Study Island 
students may not have kept pace with the non-Study Island participants. 

A correlation and regression analysis was performed using Study Island and 
NOCTI and although positive, was insignificant. The R“ value was also low at 1.2 
percent. Therefore, there is a minimal relationship between the two and nothing can be 
inferred about how the students performed on the post-NOCTI test. (See table 29) 

An analysis of written NOCTI over the last three years shows significant 
improvements overall but the largest improvements occurred between the 2009/2010 and 
2010/2011 school years and this reflects the students who were involved in the Study 
Island process” versus those who were not. In the advanced category, the largest jump 
was between the 2009/2010 and the 2010/201 1 school years. There were 112 seniors 
who scored at an Advanced level in 2009/2010 (non-Study Island participants), while in 
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2010/2011 school year there were 155 students who scored at the Advanced level (Study 
Island participants). This shows an improvement of 43 Advanced scores versus an 
improvement of 14 for the previous school year. The Competency levels however show 
a trend of going down which may imply that those who may have scored at the 
Competency level are now scoring at the Advanced level due to Study Island. For the 
last three years the Competency scores were 36, 29, and 24 starting from the school year 
2008/2009. 

The Basic scores fell as well with a 47, 31, and 8 respectively each year, which 
may mean students who normally would score a basic, would now score at a competent 
level or even at the Advanced level. A really significant indication of improvement 
shows at the Below Basic level. Between the 2008/2009 and 2009/2010 school year the 
Below Basic score went up from 32 to 34 students (non-Study Island participants). 
Between the 2009/2010 and the 2010/2011 school years however, the Below Basic levels 
went down from 34 to 15 students; a drop of 19 students (Study Island” participants). 
(See Table 17, p. 65) 

Another point of interest is the comparison of the pre-NOCTI written 
versus the Post-NOCTI written. It is important to note that in every year, the 
improvement gap went up between the two but the largest improvement gap occurred 
between the 2009/2010 (8) and the 2010/201 1 school year (52). A paired t-test was 
performed using the Pretest and the Posttest to determine if there was a significant 
difference in the mean scores. (See Table 28) A histogram of the mean differences is 



shown in Figure 14. 
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The mean score for the Pretest (M = 61.87) and the Posttest (M = 70.12) shows a 
difference of 8.26 with a confidence interval between 6.89 and 9.62 and the p value (p = 
0.000) shows a very high significance. 



Histogram of Differences 

(with Ho and 95% t-confidence interval for the mean) 




-12 -8 

Differences 



Figure 14. Histogram of differences between mean scores of Pre-NOCTI and Post- 



NOCTI 



Conclusion 

There was a significant increase in scores from the NOCTI pretest and the NOCTI 
posttest for all seniors. The between-subject t-test indicated that that the non-Study 
Island participants scored better on the NOCTI posttest than those who participated in 
Study Island. The Factorial ANOVA that included the pre and post-NOCTI scores and 
whether the students participated in Study Island or not, indicated that the differences 
were minimal which could indicate that one could argue that the Study Island students 
may not have been able to keep pace if they did not participate in study Island. All 
descriptive statistical data indicate that the students improved in NOCTI from previous 



graduating classes. 
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Research Question 6 

To what degree does Study Island Software aid IEP students in reaching 
proficiency levels in the state end-of-program technical assessments (NOCTI) in this 
part-time CTC? 

Findings 

Study Island and NOCTI: IEP Students 

The data indicates that Study Island had little or no impact on NOCTI for students 
with IEPs in NOCTI. Students that did not participate in Study Island outscored those 
that did. The correlation and R“ values were weak. 

In terms of the NOCTI posttest for the written portion of the assessment for IEP 
seniors the mean score for IEP students on the posttest was 67.61 (SD = 12.66) while the 
mean score for non-IEP students in the posttest was 72.21 (SD = 12.41). Even though the 
non-IEP students scored higher, the difference did not prove to be significant. 

A Factorial ANOVA was performed using the NOCTI posttests differences with 
and without IEP status, and whether Study Island was used or not. The difference 
between scores on the NOCTI posttest was significant in that non-Study Island 
participants scored higher than the students who did participate in Study Island. 

The Factorial ANOVA showed no significant interaction between IEP students and Study 
Island and non-Study Island participants and vice versa. 

A correlation and regression analysis was performed using Study Island and 
NOCTI for IEP students and although positive, was insignificant. The R“ value was also 
low at 4.7 percent and a correlation of 0.095 which shows minimal relationship between 
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the two and that nothing can be inferred about how the students performed on the post- 
NOCTI test. 

The IEP students performed exceptionally in the NOCTI exam for the 2010/201 1 
school year. Of the 77 IEP students who were eligible to sit for the NOCTI, 58 scored at 
the Advanced level, 1 1 scored at the Competency level, no students scored at the Basic 
Level, and only eight students scored at the Below Basic level. This translates to 77 
percent receiving Advanced, 14 percent receiving a Competent, and nine percent received 
a Below Basic. See Figures 7 and 8, pp. 69 and 70) 

Another emerging phenomenon from this study was comparing the PSSA scores 
with the NOCTI scores for IEP students by level of achievement. Of those students who 
scored a Proficient or better in both reading and math for the PSSA Achievement test 
(17), three scored an Advanced and 14 scored a Competent on the NOCTI. Of those that 
only scored a Proficient or better in one of the categories of reading or math on the PSSA 
(13), 3 scored an Advanced and ten scored a Competent. This is the surprising part: of 
those who failed to score a Proficient or better on either reading or math of the PSSA 
(35), 6 scored a Competent and 35 scored an Advanced on the NOCTI. This data may 
show that IEP students who chose CTE are truly embedded in their course of study. And 
although Study Island was instrumental in part in helping improve 4Sight and PSSA 
scores, it may have played an even more major role in NOCTI success for IEP students. 
(See Table 20) 

Figure 15 shows the histogram of differences between the mean scores of the Pre-NOCTI 



and the Post-NOCTI Assessments. 
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Histogram of Differences 

(with Ho and 95% t-confidence interval for the mean) 
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Figure 15. Histogram of differences between mean scores of Pre-NOCTI and Post- 
NOCTI: IEP students 

Conclusion 

The data show that there was a large significant difference between the pre and 
post-NOCTI scores for the IEP students. Non IEP students scored higher than the IEP 
students but the significance was not significant which one could argue that if it were not 
for Study Island participation, the IEP students may not have been able to keep pace with 
the non IEP students. The non-Study Island participants scored better than the Study 
Island participants and this difference was significant. There was no indication of 
interaction and therefore whether having an IEP or not, and participating in Study Island 
made no difference. The regression analysis showed no indication of a good correlation 
so Study Island was not a good predictor of how students would perform on the NOCTI 
and a very weak R value which means that Study Island had no effect on the NOCTI 




score. 
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Overall Conclusion 

Table 36 is a summation of the impact that Study Island may or may not have had 
on student achievement tests taken at this suburban CTC. The statistic that showed the 
largest relationship was correlation. It appears that in every case, with the exception of 
the NOCTI for non IEP and IEP students, Study Island can be used as a prediction as to 
how these students will perform in the associated assessments. Regression analysis 
showed a positive effect in the 4Sight read for non IEP and IEP students, in the 4Sight 
math for IEP students, and in the PSSA read and math categories for IEP students. The 
Factorial ANOVA only faired positive in one category in the NOCTI exam for the IEP 
students. In this case, the IEP students who participated in Study Island showed evidence 
of keeping up with their counterparts who did not participate in Study Island. The paired 
t-test used to determine if there was a significant difference between the 4Sight pretest 
and 4Sight posttest for non IEP and IEP students showed that for whatever reason, both 
groups improved in math in at least the 4Sight test is concerned. Descriptive statistics 
were used to determine if PSSA and NOCTI scores have improved from previous years 
and for whatever reason, there was improvement in both for non IEP and IEP students. 

The data show that Study Island may be more beneficial for IEP students but not 
with over whelming results. There are 14 instances where a yes occurs in table 36 that 
are aligned with students with IEPs. There are nine instances where a yes occurs in table 
36 that are aligned with non IEP students. 
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Table 36 

Impact of Study Island on 4Sight, PSSA, and NOCTI 





Descriptive 
Means, SD 


Paired T- 
Test 


2-Sample 

T-Test 


Between 
Subject T- 
Test 


Factorial 

ANOVA 


Regression 

Analysis 


Correlation 


4Sight 
Read No 
IEP 




No 




No 


No 


Yes 


Yes 


4Sight 
Math No 
IEP 




Yes 




No 


No 


No 


Yes 


4Sight 

Read 

IEP 




No 




Yes 


No 


Yes 


Yes 


4Sight 

Math 

IEP 




Yes 




No 


No 


Yes 


Yes 


PSSA 
Read No 
IEP 






No 


No 




No 


Yes 


PSSA 
Math No 
IEP 






No 


No 




No 


Yes 


PSSA 

Read 

IEP 


Yes 






No 


No 


Yes 


Yes 


PSSA 

Math 

IEP 


Yes 






No 


No 


Yes 


Yes 


NOCTI 
No IEP 


Yes 


Yes 




No 


Yes 


No 


No 


NOCTI 

IEP 


Yes 


Yes 




No 


No 


No 


No 
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Implications of the Study 

When comparing Study Island participants with non-Study Island participants, the data 
show that typically non-Study Island students outperform the students who were Study 
Island participants. When comparing IEP students with non-IEP students, the data shows 
that students without IEPs score higher on standardized achievement tests than those with 
IEPs. The study also shows that there are no interactions between students with IEPs 
and whether or not they participated in Study Island or not. 

Emerging Data 

The NOCTI Pretest has the largest impact on the success of the NOCTI Posttest. 
This was initially not part of the data collection and analysis. It would be highly 
encouraged for all CTCs have their seniors take the Pre-NOCTI Test in the Fall to 
prepare them for a successful Post-NOCTI experience. 

The second piece of emerging data was that for purposes of NOCTI, it did not 
appear relevant at what level the IEP students scored in the PSSA standardized tests, as 
the majority of these students scored Advanced and Competent in the NOCTI exam. 
Recommendations 

• Given the overall 

o Use professional development time to train teachers to teach 
literacy and numeracy strategies to create a diversified teaching 



approach in teaching academics. 
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o Bring in the MAX Teaching professionals to help with this 
diversified instruction. 

o Use math and literacy coaches to train teachers how to deliver 
academic instruction related to CTE. 
o Get involved with the PDE Tech Centers That Work Program 
through the current TAP initiative. 

o Get academic teachers involved with CTE teachers to develop co- 
teaching strategies, in that CTE teachers can teach related 
academics and academic teachers can focus in on skill-specific 
applied academics at the individual task level. 

• Have the seniors take the Pre-NOCTI in September to prepare them for the 
Post-NOCTI. 

• Track all relevant data: 

o PSSA scores taken from the PDE e-Metric System, 
o Pre-NOCTI Scores 
Recommendations for Further Research 

To better understand the impact on the students with IEPs, it may be beneficial to 
breakdown the analysis even further with regards to the specific disability; namely 
whether the student was considered as needing learning support or emotional support. A 
researcher may discover that that emotional support students may not benefit at all from 



Study Island. 
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Another opportunity for research is to consider if students did better on Study 
Island while working from home or at school. It may be determined that within the 
tranquility of their own home, students may perform better. 

One may want to see if pulling students out of their major study in CTE has a 
negative impact on attitudes towards CTE or Study Island. A student may resent the fact 
that they are being pulled away to do something they deem to have no added value in for 
CTC skill set. 

One might also study the environment in which Study Island takes place at 
school to measure distractions, whether or not the monitor is of any help, or if there 
should be a math or reading expert in the room. A student may show that students have 
more respect for one who is highly qualified in the area of study and may feel a higher 
comfort level with this type of support. 

Given the primarily monoculture of this study, it may be beneficial to extend this 
study to a more multicultural group such as you might find in an inner-city CTC. This 
suburban CTC is made up primarily of Caucasian, middle class students with a few 
students of Hispanic, African American, and Indian backgrounds, and a small 
economically disadvantaged representation. 

Study Island facilitates a lot of data regarding time on task. It could be beneficial 
to track time on task versus outcome to determine if the amount of time spent on a 
subject correlates with the success of the testing. The researcher of this study avoided 
this as it may have been thought that time ticking away while students were logged on 
might not always represent “authentic” work. A student who is logged on to Study Island 
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but is daydreaming won’t give you a measurement of actual productive work. If there 
were a way to monitor this, such as when the student is working one on one with a 
facilitator who can verify “on task” activity, this may add another valuable dimension to 
the study. 

The number of items completed may be another valuable measure that could be 
correlated with outcome. Would completion of more items by the students correlate 
directly with higher scores on tests? This is a quality versus a quantity issue in which the 
researcher would be more valuable for a student to spend more time on an item rather 
than rush through the exercises to see how many he/she could get done. If this is 
something that can be monitored by a facilitator it may provide valuable data. 

Summary 

This study provides CTE educational leaders with additional knowledge, relating 
to the importance of integrating academics into CTE. Also should at justify all that 
educational leaders are doing today to improve PSSA and NOCTI scores. A vehicle such 
as Study Island, as evidenced by the data in this research, does not seem to be a valued 
integral part of an overall comprehensive plan for students without IEPs to make these 
improvements. However, for students with IEPs, the data shows that Study Island does 
show sevidence of some impact. Because this program is tailored to the PSSAs, it may 
be more beneficial in a purely academic setting. The state is moving away from PSSAs 
and towards the Keystone exams as part of the new high school graduation requirements 
to better help students prepare for college and career. The school of this study may be 
better served by giving attention and funding dollars towards improvement of NOCTI 
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exams, industry specific certifications, and articulation agreements with local colleges 
and postsecondary trade and industrial schools. 

Starting with the class of 2015, new high school graduation requirements 
will help ensure that Pennsylvania’s students prepared for college and 
career. Along with current requirements (course completion and grades, 
completion of culminating project, and demonstration of proficiency in 
each of the state standards not assessed by a state assessment) students 
will complete one of the following pathways for each main subject: 

1. Successful completion of courses in which a Keystone Exam serves 
as the final exam and counts for at least one-third of the course mark; 
or 

2. Demonstration of proficiency on independently-validated local 
Assessment systems; or 

3. Demonstration of proficiency on a Keystone Exam used as a 
Stand-alone graduation requirement in a given content area; or 

4. Demonstration of proficiency on an Advanced Placement or 
the International Baccalaureate Exam. 

(Pennsylvania Department of Education SAS, n.d., p. 1) 

As graduation requirements become more stringent and defined by the year 2015, 
it will be crucial for CTE educators to get involved in academic as well as career- 
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education initiatives such as those examined in this document in order to ensure CTE 
student success. 

Several requirements dictate the testing of career and technical 
education students. First, at their November 1996 meeting, the 
State Board of Education approved an occupational competency 
measure requiring the use of student occupational competency 
assessment for all secondary and adult career and technical 
education completers. Second, the Carl D. Perkins Career and 
education Act of 2006 requires that each state develop a system 
of core performance measures and standards for the purpose of 
evaluating its secondary, adult, and postsecondary career and 
technical education programs. Third, Pennsylvania is a unified 
state where state and federal career and technical programs are part 
of the Governor’s Executive Order for the Pennsylvania Workforce 
Investment Board which calls for an integrated workforce 
Investment system with core performance measures and 
Standards. (Bureau of Career and Technical Education, 2007, p. 2) 
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